{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Strategies for reducing computational load\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial covers two strategies for pruning the computational load of TPOT to decrease run time."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Successive Halving\n",
    "\n",
    "This idea was first tested with TPOT by Parmentier et al. in [\"TPOT-SH: a Faster Optimization Algorithm to Solve the AutoML Problem on Large Datasets\"](https://www.researchgate.net/profile/Laurent-Parmentier-4/publication/339263193_TPOT-SH_A_Faster_Optimization_Algorithm_to_Solve_the_AutoML_Problem_on_Large_Datasets/links/5e5fd8b8a6fdccbeba1c6a56/TPOT-SH-A-Faster-Optimization-Algorithm-to-Solve-the-AutoML-Problem-on-Large-Datasets.pdf). The algorithm operates in two stages. Initially, it trains early generations using a small data subset and a large population size. Later generations then evaluate a smaller set of promising pipelines on larger, or even full, data portions. This approach rapidly identifies top-performing pipeline configurations through initial rough evaluations, followed by more comprehensive assessments. More information on this strategy in Tutorial 8.\n",
    "\n",
    "In this tutorial, we will cover the following parameters:\n",
    "\n",
    "`population_size`\n",
    "\n",
    "`initial_population_size`\n",
    "\n",
    "`population_scaling`\n",
    "\n",
    "`generations_until_end_population`\n",
    "\n",
    "`budget_range`\n",
    "\n",
    "`generations_until_end_budget`\n",
    "\n",
    "`budget_scaling`\n",
    "\n",
    "`stepwise_steps`\n",
    "\n",
    "Population size is the number of individuals evaluated each generation. Budget refers to the proportion of data to sample. By manipulating these parameters, we can control how quickly the budget increases and how population size changes over time. Most often, this will be used to start the algorithm by evaluating a large number of pipelines on small subsets of the data to quickly narrow now best models, before later getting a better estimate with larger samples on fewer datasets. This can reduce overall computational cost by not spending as much time evaluating poor performing pipelines.\n",
    "\n",
    "`population_size` determines the number of individuals to evalaute each generation. Sometimes we may want to evaluate more or fewer individuals in the earlier generations. The `initial_population_size` parameter specifies the starting size of the population. The population size will gradually move from `initial_population_size` to `population_size` over the course of `generations_until_end_population` generations. `population_scaling` dictates how fast that scaling takes place. The interpolation over `generations_until_end_population` is done stepwise with the number of steps specified by `stepwise_steps`.\n",
    "\n",
    "The same process goes for the budget scaling. \n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell illustrates how the population size and budget change over time with the given settings. (Note that tpot happens to converge on this dataset fairly quickly, but we turn off early stop to get the full run. )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAw8AAAGwCAYAAADv31lfAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAATNxJREFUeJzt3Qd4VNX28OGVkAIJvYVeBASjUgNIUZBiRFQQ9AIqIPUiRYqocJWAgKKoNEFQQbCLWOAqCHKRcpVAqEoXECkBAlwpSSghZL5nbf8z3yQkcAIzmZnM732eIXPOTCY7JzOcvc5ee+0Am81mEwAAAAC4jsDrPQEAAAAAFMEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlQdaelrulpqbKli1bJCIiQgIDiacAAPAFaWlpkpCQIHXq1JGgILo0QE7gkyZiAocGDRp4uhkAAOAGxMXFSf369T3dDMAvEDyImBEH+38+pUuX9nRzAACABceOHTMX/+zncQDuR/CgEz/+L1VJA4dy5cp5ujkAACAbSDkGcg6fNgAAAACWEDwAAAAAsITgAQAAAIAlBA8AAAAALCF4AAAAAGAJwQMAAAAASwgeAAAAAFhC8AAAAADAEoIHAAAAAJYQPAAAAADw/uBhzZo18tBDD0mZMmUkICBAFi5cmO5xm80mMTExUrp0acmXL5+0atVK9u7dm+45f/31lzzxxBNSsGBBKVy4sPTq1UuSkpJy+DcBAADe7nr9jsysWrVK6tatK6GhoVK1alWZN29ejrQV8FYeDR6Sk5OlVq1aMmPGjEwfnzhxokybNk1mzZol69evl/DwcImOjpaLFy86nqOBw44dO2T58uXy/fffm/8Y+vbtm4O/BQAA8AXX63dkdODAAWnbtq3ce++9snXrVhkyZIj07t1bli1b5va2At4qwKaX972AXgH49ttvpX379mZbm6VXBp599lkZPny42Xf27FmJiIgwUX/nzp1l165dEhkZKRs2bJCoqCjznKVLl8oDDzwgR44cMd9vhT63fPnycvjwYSlXrpxLfh9t/4XLV1zyWvhbvuA85n0CADdFT3vnz3u6Ff4nLExP9i59yZs5f2fsd2TmhRdekMWLF8v27dsd+7T/cebMGdPfAPxRkHgpjfaPHz9uUpXsChUqJA0bNpTY2Fjz4dWvmqpkDxyUPj8wMNCMVDzyyCOZvvalS5fMzS4xMdHl7dfAITKGKxOuFFWxiCzo14gAAsDNBQ5Nm4qsXevplvgfTSkOD3fLS+t5/Ny5c45tTTHS283SfoZzP0RpBoSOQAD+ymsnTGvgoHSkwZlu2x/TryVLlkz3eFBQkBQtWtTxnMxMmDDBBCL2m45ewPttPHia0RwAN0dHHAgcch09jzuf1/U87wral8isH6KByoULF1zyMwBf47UjD+40cuRIGTZsmGM7Pj7e5QGEptjsHBvt0tf0V+dTrkjU+P94uhkAcpuEBLddCUcWaUtusnPnTilbtqxj2xWjDgB8LHgoVaqU+ZqQkGCqLdnpdu3atR3POXHiRLrvS01NNRWY7N+fmYzDmc5Dna6iqTVhIV57eAEAGjgQPOQKBQoUMFUXXU37EtrvcKbb+rO0CiTgj7w2baly5crmQ7tixYp0nXydy9CoUSOzrV910tKmTZscz/npp58kLS3NzI0AAAC4UdrPcO6HKK3uaO+HAP7Io5fGdT2Gffv2pZskraXQdM5ChQoVzISk8ePHS7Vq1UwwMWrUKFNByV4Z4bbbbpP7779f+vTpY8q5Xr58WQYOHGgmU1uttAQAAPzD9fodmtasqcwfffSRebxfv34yffp0ef7556Vnz57mAuWXX35pKjAB/sqjwcPGjRtN7WQ7+zyE7t27m3Ks+mHVmsy6boOOMDRt2tSURsubN6/jez799FMTMLRs2dJUWerYsaNZGwIAACA7/Y5jx47JoUOHHI/rhUsNFIYOHSpTp0415WBnz55tKi4B/spr1nnwJHes8wDXOZ+S6ih7q5PQmUsC4IYlJ4vkz+/20qHIGZy/gZzntXMeAAAAAHgXggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlBA8AAAAALCF4AAAAAGAJwQMAAAAASwgeAAAAAFhC8AAAAADAEoIHAAAAAJYQPAAAAACwhOABAAAAgCUEDwAAAAAsIXgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlBA8AAAAAckfwkJiYKEOGDJGKFStKvnz5pHHjxrJhwwbH4zabTWJiYqR06dLm8VatWsnevXs92mYAAOCdZsyYIZUqVZK8efNKw4YNJS4uLsvnXr58WcaOHStVqlQxz69Vq5YsXbo0R9sLeBuvDx569+4ty5cvl48//li2bdsm9913nwkQ4uPjzeMTJ06UadOmyaxZs2T9+vUSHh4u0dHRcvHiRU83HQAAeJH58+fLsGHDZPTo0bJ582YTDGif4cSJE5k+/6WXXpJ3331X3n77bdm5c6f069dPHnnkEdmyZUuOtx3wFgE2vXTvpS5cuCAFChSQRYsWSdu2bR3769WrJ23atJFx48ZJmTJl5Nlnn5Xhw4ebx86ePSsREREyb9486dy5s6Wfc+TIESlfvrwcPnxYypUr57bfBzfmfEqqRMYsM/d3jo2WsJAgTzcJgK9KThbJn//v+0lJIuHhnm4RbkJ2z9860lC/fn2ZPn262U5LSzPfP2jQIBkxYsRVz9c+xosvvigDBgxw7OvYsaPJdPjkk09c/NsAvsGrRx5SU1PlypUrZqjQmX5of/75Zzlw4IAcP37cjETYFSpUyPznEBsbm+XrXrp0Sc6dO+e4aWoUAADwTXoedz6v63k+o5SUFNm0aVO6PkNgYKDZzqrPoK+TVR8E8FdeHTzoqEOjRo3MCMPRo0dNIKGRvn7Ijx07ZgIHpSMNznTb/lhmJkyYYIIM+y0yMtLtvwsAAHAPPY87n9f1PJ/RqVOnTD8iO30GTWmaNGmSmUupoxSaRv3NN9+YPgjgr7w6eFA610Ezq8qWLSuhoaFmfkOXLl3M1YIbNXLkSJPeZL9pHiMAAPBNeh53Pq/red4Vpk6dKtWqVZMaNWpISEiIDBw4UHr06HFTfRDA13n9u18rHKxevVqSkpJMTqNWRdDqB7fccouUKlXKPCchISHd9+i2/bHMaBBSsGBBx01HOAAAgG/S87jzeV3P8xkVL15c8uTJk60+Q4kSJWThwoWSnJwsBw8elN27d0v+/PlNHwTwV14fPNhpFSUtx3r69GlZtmyZtGvXTipXrmw+8CtWrHA8T3MdteqSpjsBAAAoHTnQgivOfQZNRdLt6/UZdN6DZkDoXMyvv/7a9EEAf+X1ZWs0UNC0perVq8u+ffvkueeeM8OHOmwYEBBg1oAYP368GVbUYGLUqFGmOkL79u093XQAAOBFtExr9+7dJSoqSho0aCBTpkwxowrap1DdunUzQYJ9zoRejNTS8LVr1zZfx4wZYwKO559/3sO/CeA5Xh882HMXtRxb0aJFTYm0V155RYKDg83j+gHWD37fvn3lzJkz0rRpU7OAS8bqCAAAwL916tRJTp48aRaX1UnSGhRon8E+ifrQoUPp5jPomlG61sMff/xh0pUeeOABMxezcOHCHvwtAM/y6nUecgrrPHg31nkA4DKs85CrcP4Gcp7PzHkAAAAA4FkEDwAAAAAsIXgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlBA8AAAAALCF4AAAAAGAJwQMAAAAASwgeAAAAAFhC8AAAAADAEoIHAAAAAJYQPAAAAACwhOABAAAAgCUEDwAAAAAsCbL2NMA7nE+54ukm+I18wXkkICDA083wDzabyPnznm6Ff0hO9nQLAMCnETzAp0SN/4+nm+A3oioWkQX9GhFA5ETg0LSpyNq1nm4JAADXRdoSfOIKuHZkkbM2HjwtFy4z0uN2OuJA4JDzmjQRCQvzdCsAwOcw8gCvp1e+9Qo4HdmcSw1jhMdDEhJEwsM93Qr/oIEDo2oAkG0ED/CZACIshLcrcjkNHAgeAABejLQlAAAAAJYQPAAAAACwhOABAAAAgCUEDwAAAAAsIXgAAAAAYAnBAwAAAADfDx6uXLkio0aNksqVK0u+fPmkSpUqMm7cOLHpiqz/R+/HxMRI6dKlzXNatWole/fu9Wi7AQCAd5oxY4ZUqlRJ8ubNKw0bNpS4uLhrPn/KlClSvXp108coX768DB06VC5evJhj7QW8jVcHD6+//rrMnDlTpk+fLrt27TLbEydOlLffftvxHN2eNm2azJo1S9avXy/h4eESHR3NBxsAAKQzf/58GTZsmIwePVo2b94stWrVMn2GEydOZPr8zz77TEaMGGGer/2QOXPmmNf417/+leNtB7yFVwcPa9eulXbt2knbtm3NVYJHH31U7rvvPsdVAh110CsCL730knlezZo15aOPPpKjR4/KwoULPd18AADgRSZNmiR9+vSRHj16SGRkpLnwGBYWJh988EGW/ZAmTZrI448/bvoh2gfp0qXLdUcrgNzMq4OHxo0by4oVK+T3338327/++qv8/PPP0qZNG7N94MABOX78uElVsitUqJAZhoyNjc3ydS9duiTnzp1z3BITE3PgtwEAAO6g53Hn87qe5zNKSUmRTZs2peszBAYGmu2s+gzaD9HvsQcLf/zxhyxZskQeeOABN/42gHcLEi+mQ4X6n0CNGjUkT548Zg7EK6+8Ik888YR5XAMHFRERke77dNv+WGYmTJggL7/8sptbDwAAcoKOIjjTNKMxY8ak23fq1CnTj8isz7B79+5MX1dHHPT7mjZtarIdUlNTpV+/fqQtwa959cjDl19+KZ9++qnJOdTcxA8//FDefPNN8/VmjBw5Us6ePeu47dy502VtBgAAOUvP487ndT3Pu8KqVavk1VdflXfeecf0Q7755htZvHixKd4C+CuvHnl47rnnzOhD586dzfadd94pBw8eNCMH3bt3l1KlSpn9CQkJptqSnW7Xrl07y9cNDQ01Nzsd3QAAAL6pQIECUrBgwWs+p3jx4iaLQfsIznTb3p/ISCs+du3aVXr37u3ohyQnJ0vfvn3lxRdfNGlPgL/x6nf9+fPnr/pg6gc/LS3N3NcSrvqB13kRzoGAVl1q1KhRjrcXAAB4p5CQEKlXr166PoP2J3Q7qz5DVv0Q5Vw2HvAnXj3y8NBDD5k5DhUqVJDbb79dtmzZYiol9OzZ0zweEBAgQ4YMkfHjx0u1atVMMKFXCcqUKSPt27f3dPMBAIAX0TKtmrkQFRUlDRo0MBUbdSRBqy+pbt26SdmyZU2Gg70fov2OOnXqmGIs+/btM/0M3W8PIgB/49XBg67noB/S/v37mxrMGhT885//NIvC2T3//POOIcQzZ86YSU1Lly41i78AAADYderUSU6ePGn6EVpYRVOctc9gn0R96NChdCMNWgpeL1Tq1/j4eClRooTjwibgrwJsjLvJkSNHzKqRhw8flnLlynm6OYBHnU9JlciYZeb+zrHREhbi1dcYfF9yskj+/H/fT0oSCQ/3dIsAn8H5G8h5Xj3nAQAAAID3IHgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAADuCx72798vL730knTp0kVOnDhh9v3www+yY8eOG3k5AAAAALkxeFi9erXceeedsn79evnmm28kKSnJ7P/1119l9OjR7mgjAAAAAF8MHkaMGCHjx4+X5cuXS0hIiGN/ixYtZN26da5uHwAAAABfDR62bdsmjzzyyFX7S5YsKadOnXJVuwAAAAD4evBQuHBhOXbs2FX7t2zZImXLlnVVuwAAAAD4evDQuXNneeGFF+T48eMSEBAgaWlp8ssvv8jw4cOlW7du7mklAAAAAN8LHl599VWpUaOGlC9f3kyWjoyMlHvuuUcaN25sKjABAAAAyJ2CsvsNOkn6/fffl5iYGDP/QQOIOnXqSLVq1dzTQgAAAAC+GTysWbPGMfKgN7vLly9LbGysGYUAAAAAkPtkO22pefPmUqtWravKsv71119y7733urJtAAAAAHx9hWmdNN2yZUuZN29euv02m81V7QIAAADg68GDVlgaOXKkfPzxxzJw4EAZNmyYI2jQxwAAAADkTtkOHuyBQocOHeS///2vfPXVV9KmTRs5c+aMO9oHAAAAwJfTluy0ylJcXJwJHDSNCQAAAEDule3goXv37pIvXz7HdqlSpWT16tUmeKhQoYKr2wcAAADAV0u1zp0796p9oaGh8uGHH7qqTQAAAAB8NXj47bff5I477pDAwEBz/1pq1qzpqrYBAAAA8LXgoXbt2nL8+HEpWbKkua9VlZzLstq39euVK1fc2V4AAAAA3hw8HDhwQEqUKOG4DwAAAMD/WAoeKlasmOl9AAAAAP4j29WWdGL04sWLHdvPP/+8FC5cWBo3biwHDx50dfsAAAAA+Grw8OqrrzpKtcbGxsr06dNl4sSJUrx4cRk6dKg72ggAAADAF0u1Hj58WKpWrWruL1y4UB599FHp27evNGnSRJo3b+6ONgIAAADwxZGH/Pnzy//+9z9z/8cff5TWrVub+3nz5pULFy64vIGVKlUyVZwy3gYMGGAev3jxorlfrFgx07aOHTtKQkKCy9sBAAB834wZM0zfQvstDRs2lLi4uCyfqxdFM+uDtG3bNkfbDPh08KDBQu/evc3t999/lwceeMDs37Fjh/kwutqGDRvk2LFjjtvy5cvN/scee8x81VSp7777ThYsWGBWuj569Kh06NDB5e0AAAC+bf78+TJs2DAZPXq0bN68WWrVqiXR0dFy4sSJTJ//zTffpOuDbN++XfLkyePogwD+KPBGIvZGjRrJyZMn5euvvzZX/NWmTZukS5cuLm+glogtVaqU4/b9999LlSpVpFmzZnL27FmZM2eOTJo0SVq0aCH16tUzK2CvXbtW1q1b5/K2AAAA36X9hT59+kiPHj0kMjJSZs2aJWFhYfLBBx9k+vyiRYum64PoBUx9PsED/Fm25zxoZSWdJJ3Ryy+/LO6WkpIin3zyiblqoMOGGrBcvnxZWrVq5XhOjRo1pEKFCmYy91133ZXp61y6dMnc7BITE93edgAA4B56Hj937pxjOzQ01Nwy9iG03zBy5EjHvsDAQNOH0D6DFXrBsnPnzhIeHu7C1gO5fOTBk3SC9pkzZ+Spp54y27rqdUhIiAlonEVERJjHsjJhwgQpVKiQ46ZXHwAAgG/S87jzeV3P8xmdOnVKrly5YvoI2ekz2OncCE1b0rRtwJ9le+TBkzTib9OmjZQpU+amXkevOujohV18fDwBBAAAPmrnzp1StmxZx3bGUQdX9UHuvPNOadCggctfG/AlPhM86AJ0//nPf8zkJTvNP9RhSB2NcB590GpL+lhWMg5nOg91AgAA31KgQAEpWLDgNZ+j61HpZOeMFRmv12dQycnJ8sUXX8jYsWNd0l7Al/lM2pJOhC5ZsmS68mg6QTo4OFhWrFjh2Ldnzx45dOiQmdQNAACgNM1Z+w3OfYa0tDSzfb0+g1Z01LmSTz75ZLZ/rgYsmVVz0rL3+hjga3xi5EE/3Bo8dO/eXYKC/n+TNa+xV69eJgVJKyLoVYdBgwaZ/wSymiwNAAD8k/YXtC8RFRVl0o+mTJliRhW0+pLq1q2bSX/KOGdCU5bat2/vqDCZHTabLdP9GoxoQAPk+uBBh/eGDx9uInWNpDN+KHQykqtpupKOJvTs2fOqxyZPnmyqJejicPpB1HrN77zzjsvbAAAAfFunTp1MqfmYmBgzSbp27dqydOlSxyRq7Wton8KZZjT8/PPPZmHc7Jg2bZr5qtUhZ8+ebRayde4rrVmzxlSIBHxNgC2rkDgLOmFZP1wDBw6U0qVLmw+Fs3bt2omvOXLkiJQvX14OHz4s5cqV83RzAI86n5IqkTHLzP2dY6MlLMQnBih9V3KyiL1TkZQkQglIIFecvytXruyYs6ltc05R0hEHXVhX51DoKteAL8l2r0Cj7//+978mWgcAAMDVDhw4YL7ee++9pthLkSJFPN0kwDMTpjXCz+ZgBQAAgF9auXKlCRy0OqSmQKWmpnq6SUDOBg86uWjEiBHy559/3txPBgAAyOUuXLhgiruEhYXJ7bffblK/lRZ4ee211zzdPMD9wYNONlq1apVUqVLF1FXWKkfONwAAAPxNL7j++uuvpu+UN29ex/5WrVrJ/PnzPdo2IEfmPOjIAwAAAK5v4cKFJkjQEvLORWZ0FGL//v0ebRuQI8GD1kcGAADA9WlpWF3kNiNdXyJjxUrAF9xQDUatT6yR9K5duxzR88MPP8xKiQAAAE50QbrFixebOQ7KHjDo2g/XW9kayBXBw759++SBBx6Q+Ph4qV69utmnKzFqFSb9cOhcCAAAAIi8+uqrZo2snTt3mkpLU6dONffXrl0rq1ev9nTzAPdPmH7mmWdMgKALsmzevNnctHKALoaijwEAAOBvTZs2la1bt5rA4c477zQrVWsaU2xsrNSrV8/TzQPcP/KgUfK6devSVVYqVqyYKTfWpEmT7LcAAAAgF9OLru+//76nmwF4JngIDQ2VxMTEq/YnJSWZ5dYBAAD82blz5yw/t2DBgm5tC+Dx4OHBBx+Uvn37ypw5c6RBgwZm3/r166Vfv35m0jQAAIA/K1y4sOVKSlqEBsjVwcO0adNMuVatEBAcHGz2aR6fBg46CQgAAMCfrVy50nH/zz//NAvFPfXUU47qSjrf4cMPPzQFZ4BcHzxoNL1o0SLZu3ev7N692+y77bbbpGrVqu5oHwAAgE9p1qyZ4/7YsWNl0qRJ0qVLF8c+veCqk6ffe+891s+Cf6zzoKpVq2ZuAAAAyJyOMsyaNSvT9R969+7tkTYBbg8ehg0bJuPGjZPw8HBz/1o0ugbg42w2yXf5kpw/fU4khMUf3So5WcI83QYAbqPrYGmlpYkTJ6bbr4vE6WNArgwetmzZIpcvX3bcB5CL2Wzy1afPS1T8LpHJnm6Mf7HZbGJtiiUAXzF58mTp2LGj/PDDD9KwYUOzLy4uzqR/f/31155uHpBtATY9W/m5I0eOmOhfF74rV66cp5sDeJQtKUkCChTwdDP8zoaykXL7/q0SFvp3IQoAuef8re2bOXNmurmiWqWSkQf4xZyHnj17mqpKBTJ0LpKTk2XQoEHywQcfuLJ9AHKYc3nB84fjRcLDPdqe3O58yhW5+/WVciE4VHZaLO0IwLdokPDqq696uhmAZ4IHLS2mq0lnDB4uXLggH330EcEDkIuEFSlE8OBuKalyISSvp1sBwE1+++23LC/U5M2bVypUqGAW4AVyXfCgqyVqhpPedIVpfcM7L3CyZMkSKVmypLvaCQAA4HNq167tGNG1Z4o7j/DqmlmdOnWSd999N13fCvBWgdlZ36Fo0aLmDX/rrbdKkSJFHLfixYubdKYBAwa4t7UAAAA+5NtvvzWl7XVNh19//dXc9H716tXls88+kzlz5shPP/0kL730kqebCrh25EFXS9SIuUWLFqY6gAYSdiEhIVKxYkUpU6aM1ZcDAADI9V555RUzVzQ6OtqxTxeI0wneo0aNMpWXtBT+s88+K2+++aZH2wq4NHiwr5Z44MABM/EnMNDyoAUAAIBf2rZtm7nAmpHu08fsqU3Hjh3zQOuAHJgwbf8AnD9/Xg4dOiQpKSnpHq9Zs+YNNAMAACD3qVGjhik0o6lKmqmhdO0s3aePqfj4eImIiPBwSwE3BQ8nT56UHj16mMVOMqOTpwEAACAyY8YMefjhh02akv0Cq444aH/p+++/N9t//PGH9O/f38MtBdwUPAwZMkTOnDkj69evl+bNm5uJQAkJCTJ+/Hh56623svtyAAAAuVbjxo1Nyvenn34qv//+u9n32GOPyeOPP+4oe9+1a1cPtxJwY/CgFQEWLVokUVFRZt6DpjG1bt1aChYsKBMmTJC2bdtm9yUBAAByLQ0SdEVpwC+DB11J2r6eg5Zp1TQmLd2qlQM2b97sjjYCAAD4JF1A91q6deuWY20BPBI8aF3iPXv2SKVKlaRWrVpmURO9P2vWLCldurRLGgUAAJAbDB48ON22TpbWojM6eTosLIzgAbk/eNAPgb2c2OjRo+X+++83eXz6IZg3b5472ggAAOCTTp8+fdW+vXv3ytNPPy3PPfecR9oE5Gjw8OSTTzru16tXTw4ePCi7d++WChUqmJWmAQAAkDVdcVpLtWqfSvtQQK4OHjLSIbe6deu6pjUAAAB+ICgoSI4ePerpZgDuCR6GDRtm+QUnTZqU/VYAAADkQv/+97/TbdtsNpP+PX36dGnSpInH2gW4NXjYsmWLpRcLCAi44YYAAADkNu3bt7+qr1SiRAlp0aIF62Mh9wYPK1eudH9LAAAAcpm0tLSr7us6WYCv4t0LAADgRnPmzJE77rhD8uXLZ256f/bs2Z5uFpAzwcO9995rhtqyurlafHy8qUZQrFgx84HTxeg2btyYLncwJibGrDGhj7dq1cqUQAMAAMhoxowZZn2qvHnzSsOGDSUuLu6azz9z5owMGDDA9DNCQ0PNwrhLliyx/PO0j6Jl7h966CFZsGCBuen9oUOHmseAXF9tqXbt2lctdrJ161bZvn27dO/e3eW1kXUykQYsP/zwg8kR1MBAV7a2mzhxokybNk0+/PBDqVy5sowaNUqio6Nl586d5j8GAAAANX/+fFMERhe21cBhypQpps+gi9+WLFnyquenpKRI69atzWNfffWVlC1b1pSoL1y4sOWfOXPmTHn//felS5cujn0PP/yw1KxZUwYNGiRjx4512e8HeGXwMHny5Ez3jxkzRpKSksSVXn/9dSlfvrzMnTvXsU8DBOdRB/3gv/TSS9KuXTvHMvARERGycOFC6dy5s0vbAwAAfJdWhOzTp4/06NHDbGsQsXjxYvnggw9kxIgRVz1f9//111+ydu1aCQ4ONvt01CI79CJrVFTUVft1razU1NQb/l0An13nwU5Tixo0aCBvvvmmS8ub6RWBxx57TFavXm0i/v79+5sPvjpw4IAcP37cpCrZFSpUyFxNiI2NzTJ4uHTpkrnZJSYmuqzNAAAgZ+l5/Ny5c45tTS/SW8ZRhE2bNsnIkSMd+3TisvYhtM+QVT+kUaNGJm1p0aJFJgPi8ccflxdeeEHy5MljqW1du3Y1ow8ZS9m/99578sQTT1j+Ha9cuWICEcAdNDi2+p52WfCgHzxXpwn98ccf5gOnQ4z/+te/ZMOGDfLMM89ISEiISZHSwEHpSIMz3bY/lpkJEybIyy+/7NK2AgAAz4iMjEy3PXr0aJMR4ezUqVOmA55ZnyGrVZ61H/LTTz+ZTr7Oc9i3b5+5iKmdeP0ZVtbH0tKsOjn6xx9/lLvuusvsW79+vRw6dEi6det23d9Nsyy0T6NzLwB30nS8UqVKXXfphWwHDx06dMh0sROdxKzzDVxJS5rpUN+rr75qtuvUqWPmVugw483Mr9CrDs4fbJ2UnfE/HgAA4Bt0nqNmJ9hlHHW4mX6IznfQUQK9KqupRtpneOONN64ZPGRcH0u/T+3fv998LV68uLnt2LHjum2wBw7ajrCwMNbUgstpX/78+fNy4sQJs63FAVwaPGhakDMd8qtevbqZ8HPfffeJK2njM3bqb7vtNvn666/NfY2OVEJCQrpfVLczTux2lnE403moEwAA+JYCBQpIwYIFr/kc7axrAKB9BGe6be9PZKR9i4zpHNoP0Q69pkFpJoQ718fSkRJ74KBVJwF30YqlSgMIfb9dK4Up28GD8+Rld9NKS1oBwdnvv/8uFStWdEye1g/8ihUrHMGCBgI6HPj000/nWDsBAIB3046+jgBon8G+6rOOLOj2wIEDs+yHfPbZZ+Z59oXdtB+iQUVWgYMr2ec46IgD4G7295m+764VPNzwInGapvTxxx+bm05Acgetgbxu3TqTtqR5hvoB1qFDnbikdOhuyJAhMn78eDOpadu2bSZ/sEyZMlctBw8AAPybpixr2VQt775r1y5zoTE5OdlRfUn7EM4TqvVxrbak6zRo0KCVmbRPYu+H5BRSleBN77NsjzwcOXLE1Cr+5ZdfHHWOdUitcePG8sUXX0i5cuXEVerXry/ffvut+SBrWpSONGhpVufqBM8//7z54Pft29e0o2nTprJ06VLWeAAAAOl06tRJTp48aRZn09QjzVrQPoN9ErVOYraPMCgtF79s2TJzMVPXZdB5FRpIaLUlwF8F2HSWRDbcf//9ppOuUbvOdVCaWqRRu+Yb6ofQ12hApP9BHD582KXBD+CTkpNF8uf/+76u3RIe7ukW5WrnU1IlMmaZub9zbLSEhbisCB6Q6+X28/fFixdNWXq9eMpFUXjL+y3baUu63oKWT7UHDkrvv/3227JmzZobbzEAAADgYs2bNzdp7t7yOjfqzz//NKlFW7duFU/K9iUujfAzW6REKwLoXAMAAADAV61atUruvfdeOX36tCNFX33zzTeOlcY9QfvgujyCVg7zpGyPPGht40GDBpkJ03Z6X3MAXbm6NAAAAOAtihYtasoCe4pWQNIqo0FBQb4VPDz11FNmuKRhw4aO9RL0/ubNm6Vnz57mwNpvAAAAcPGCXimpHrllZ5qspvhoCVy96RpherVcFxN2fg29sq8VrooUKWLKhLZp00b27t3reHzevHnmyv/ChQulWrVqJg8/OjrazHFx7pdmrLCpqUX687OilUJ1EWINBLQz/vjjjzsWSNPUIB11UNouTRPSn5FZ2tJpi+3XSfe6Pkj+/PnN3GEdPciKvqYWBipRooRZe0F/b/syCRnTlrRdup3xpiMn6tKlSzJ8+HAz0T88PNz01+2P3Yxshy5a7QgAAAA578LlK44iCzktu0UdtLhOr169JC4uzmSpaGXMChUqSJ8+fRydX+1sa7l9LbqjVaweeOABs2K4PT1IVz5+5ZVX5KOPPjJra/Tv3186d+5sqn7eKE2/HzdunJmzq0GDlvDVtixZssSkBulixB07djQFgbRd9gXUMnrKYvs1M0cDFq3k9eSTT5oO/aeffprpa2qApd//ww8/mIBLlyq4cOFCps+dOnWqvPbaa45tvf/5559LjRo1zLYGbvpaWg1VpxZoBVMNXnRpAw1Kcix46N69+w3/MAAAAPgH7YhPnjzZXA3Xjrp2WnVbgwd7p1uDAC33r7RDrd+jIw2PPfaYo6M/ffp0c9XcHpDoVXwNSBo0aHBD7dJMGbtbbrlFpk2bZpYHSEpKMqMD9uwZXWnZec6Ds73ZaP+sWbOkSpUqjg69Lj+QFS0XXKdOHTMyoipVqpTlc3VER2/2+Rjvvvuu/Oc//zGjKfo6OmKhX+1zkjVo0aqoul/XK7lRN5Q0pZOj9cDoAivq9ttvl4cffviaq9EBAADg5uQLzmNGADz1s7PjrrvuSrfwWKNGjeStt94y/UjtQ2ruvj0oUMWKFTNBhr1/qfQ52rG306vq2qHX59xo8KCLG48ZM0Z+/fVXkyakK4gr7WhHRkZaeo1dFtuv6Uz2wEHp6uT2FKnM6MKEOuqh0wHuu+8+k5JlD06ysmXLFunatasJsnRVdKWBmh7nW2+9Nd1zNZVJ23kzsh086PCJDsnEx8c7yrVOmDDBRFq68qLzAQIAAIDraGec9WD+P00FyjgXI7OqoHa6sLDOm9CbjhTo3AINGnQ7JSXF5e0LzlCdSf9+15o7ovMmDh48aFKoli9fLi1btjQrmmdVlEgXO9QL+L179zYpYnY6iqIX9TVQynhxX0dXcnTC9DPPPGMCBJ2solGR3vSg64IS+hgAAACwfv36dNvr1q0zufbamdXUo9TU1HTP+d///mfmGThf/dfnOFf41Md1sWL9fqWd/4wTkK+1DsLu3bvNz9H5AXfffbcZycg4EqBzK5Reuc/KbRbbfyP0d9JpAp988omZa/zee+9luahbu3btzO8wadKkdI9p6pO2X3+3qlWrprtpWtPNuKFF4iZOnJiumpIOf+gfQR8DAAAA9OKyTkbWDrVO5NUFhbW0v9IgQju+Ov/h559/NilEOplYKwPpfucr97pEgHbS9Sq6TlLWdCh7ylKLFi1McKETqnUewujRo2X79u1ZtkknbGtwoG35448/zLwFnTztrGLFimaE4Pvvv5eTJ0+aq/gZVbPY/uyKiYmRRYsWmUyfHTt2mDbYA6WM/vnPf5qL+TpnQ9upoxB60xEUTVfSqk1aDUrnQ+jK0TpPRLOFNFMoR4MHLc2amJh41X49sPZIDQAAAP5NO65aKUg7+pp6o4GDVlyy04m79erVkwcffNDMh9B0Hk3XcU710TkDWsVIy6lqPr+m3MyfP9/xuKYbaYWi559/3syN0D6q/txrXdXXEqoLFiwwIwR68TtjSpAGAC+//LKMGDFCIiIizCTnzMy10P7s0r70yJEjpWbNmnLPPfeYURqtlpQZvWivoy76e+hcCvtt7dq1jvbpsXj22WfNVAOdP7FhwwYTQN2MAFt2ivb+3xtBU5XmzJnjiPo0GtTISw+g/kF8zZEjR8ycDY3eypUr5+nmAJ6VnKwJkX/f16st4eGeblGuprXT7WUXs1sGEfB3uf38rWkpesVYU8N1jQNfomsi1K5d+6ZK/GufUtdV0DQleM/7LdsjDzo0onMeNMLSF9abRoKaQ6X1ZgEAAADkTtm+xKXlsTQXS/PKtBSV5oRpLpYGDwAAAAByrxseH9eJIvaAwbmGLwAAAPzbqlWrbvo1dHK03uBdsp22pHS+wx133OFIW9L7s2fPdn3rAAAAAPjuyIOWkNJaslo2S+c9qNjYWBk6dKgpyXWtJbcBAACQPfYVkAFveJ9lO3iYOXOmvP/++9KlSxfHPl3ZTktKaUBB8AAAAHDztGynrqB89OhRU2JUt0kVh6tp4VVdG0LXitD32/WWXsh28KBLfkdFRV21X8u06kp7AAAAuHnakdOymVrLXwMIwJ10TQ1dA0Lfdy4NHrp27WpGHzIug61LZ+tKdgAAAHANvQqsHTq9QHvlyhVPNwe5lC5GFxQUZGlkK+hGJ0z/+OOPZnlw+yJxOt9BF5DTZcjtMgYYAAAAyB7t0OmqxTezcjHgKtkOHrZv3y5169Y19/fv32++Fi9e3Nz0MTty8gAAAAA/Dx5WrlzpnpYAAAAAyH3rPAAAAADwPwQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwJIbWiQOAAAAOUdXl758+bKnm+G3dIE+XYUZBA8AAABey2azyfHjx+XMmTOeborfK1y4sJQqVcrvF0ImeAAAAPBS9sChZMmSEhYW5vcdV08FcOfPn5cTJ06Y7dKlS4s/I3gAAADw0lQle+BQrFgxTzfHr+XLl898PXHihPl7+HMKExOmAQAAvJB9joOOOMDz7H+Hy34+94TgAQAAwIuRquQd+Dv8jeABAAAAgCUEDwAAAAB8P3gYM2aMGSJyvtWoUcPx+MWLF2XAgAFmElH+/PmlY8eOkpCQ4NE2AwAA7zVjxgypVKmS5M2bVxo2bChxcXFZPnfevHlX9UP0+3B9zZs3lyFDhrj0Nf/880/zN9i6datLXxe5KHhQt99+uxw7dsxx+/nnnx2PDR06VL777jtZsGCBrF69Wo4ePSodOnTwaHsBAIB3mj9/vgwbNkxGjx4tmzdvllq1akl0dLSjBGdmChYsmK4fcvDgwRxtM9xPg8kpU6Z4uhk+w+tLtQYFBZkFOTI6e/aszJkzRz777DNp0aKF2Td37ly57bbbZN26dXLXXXd5oLUAAMBbTZo0Sfr06SM9evQw27NmzZLFixfLBx98ICNGjMj0e/RKd2b9EMBfef3Iw969e6VMmTJyyy23yBNPPCGHDh0y+zdt2mRKZbVq1crxXE1pqlChgsTGxl7zNS9duiTnzp1z3BITE93+ewAAAPfQ87jzeV3P8xmlpKSYvoNzvyEwMNBsX6vfkJSUJBUrVpTy5ctLu3btZMeOHeJRNptIcrJnbvqzsyE1NVUGDhwohQoVkuLFi8uoUaPMgmv2oGzhwoVXreCsqWJ2mlJWp04dkyoWFRUlW7Zsuepn/Pvf/5Zq1aqZ59x7773y4Ycfmtd2XpFbs1buvvtus1aD/h2feeYZSdbf5//Sq3Q0SbNZ7Klp8OHgQXMR9U20dOlSmTlzphw4cMD88fU/CV1xMSQkxLzRnEVERJjHrmXChAnmjWy/RUZGuvk3AQAA7qLncefzup7nMzp16pRZdE37CVb7DdWrVzejEosWLZJPPvlE0tLSpHHjxnLkyBHxmPPnRfLn98xNf3Y2aEdeM0g0CJg6daoZ+Zk9e7al79Wg7cEHHzR/Ww36dB7s8OHD0z1H+4WPPvqotG/fXn799Vf55z//KS+++GK65+zfv1/uv/9+My/2t99+M6lrGkxoUKO++eYbKVeunIwdO9aRmgYfTltq06aN437NmjVNMKHR/5dffulY6e9GjBw50uQ82sXHxxNAAADgo3bu3Clly5Z1bIeGhrrkdRs1amRudho4aHr0u+++K+PGjXPJz8jN9Cr/5MmTzdV8DcS2bdtmtjV17Ho0LV2DNU1R11EFnQOrQdvTTz/teI7+HfR133jjDbOt97dv3y6vvPKK4zkaSGrmin3yto5STJs2TZo1a2YuTBctWtSsFl2gQAHS03JD8JCRjjLceuutsm/fPmndurUZgtRhKefRB622dL0/vv6n4vwfiw5xAgAA36QdP53YfC2aNqOdxIxVGa30G+yCg4NNGo32QzxGVzlOSvLcz84GnX/qnAakgdhbb71lRoCuZ9euXebCsXN1K+dATu3Zs0fq16+fbl+DBg3SbeuIhI44fPrpp459mjqlgYmOXGgwiFwcPOgQlg4/de3aVerVq2c+xCtWrDBDUfY3kc6JyPjmAgAA/k1TnbXvoP0GTXNR2oHUbXsKy/Vop1evnj/wwAPiMdoZDw8XX6dBhX3+g53OZXVH31HTmXSeQ0Y6Txa5LHjQ3LaHHnrIpCppGVYtraZXDbp06WJyGnv16mXSj3TISa84DBo0yAQOVFoCAAAZaZ+he/fuZvKtXqHW8pw6cdZefalbt24m/ck+Z0Lz4LVPUbVqVZPpoOkxOrm2d+/eHv5NfMP69evTbWs1TE0b0r5ciRIl0s0v0AI5553mVOiIwMcff2zW9LKPPuj3O9M0pSVLlqTbt2HDhnTbdevWNWlt+je8VmBpZTQEPjBhWnPbNFDQN8c//vEPsxicvnH0Dac0b04n0+jIwz333GOGHXXiCwAAQEadOnWSN998U2JiYqR27dpmsTEtymKfRK3ZC84d2tOnT5v8fO3I6miDpjmvXbuWeZIW6fHUgE0zQz7//HN5++23ZfDgweYxLbM/ffp0U0Fp48aN0q9fP5NRYvf444+b0Qk9/tr51yBB/3bOdERh9+7d8sILL8jvv/9u5sTaqzXZ06X0Mf2b6eiS/r01SNEJ8M6jTbrOw5o1a8wcWJ1Yj+uwwXb48GEdNzNfAb+XlKQDyX/f9D7cKvnSZVvFF743N70PwLrcfv6+cOGCbefOnearr2nWrJmtf//+tn79+tkKFixoK1KkiO1f//qXLS0tzTweHx9vu++++2zh4eG2atWq2ZYsWWIrVKiQbe7cuY7XiI2NtdWqVcsWEhJiq127tu3rr782f+8tW7Y4nrNo0SJb1apVbaGhobbmzZvbZs6caZ7jfMzi4uJsrVu3tuXPn9/8vJo1a9peeeWVdD9H9+lrXKtr7Mt/D1cK0H/Ez+kIh1YEOHz4sCnXBfg1rX2tJfmUTsrLBbm13ux8SqpExiwz93eOjZawEK/OJgW8Sm4/f2vKjk7qrVy5crqJw8iaVlrSxf/0PeFq/D3+xlkKAAAAPumdd94xFZc0tf2XX34x81KsToDHjSF4AAAAgE/SOQzjx4+Xv/76y1RPevbZZ816XnAfggcAAAD4JC2eozfkHIIH+AadmuNUwg1unvMAjzifQqnAnJIvOE+6xasAANYQPMA3AoemTUXWrvV0SwC3ihr/H083wW9EVSwiC/o1IoCAT9DF7OB5/B3+RvAA76cjDgQOOa9JE5GwME+3wi+ugGtHduPB055uil/R433h8hWqW8Gr6eJlgYGBZqFcXeNKtwl4c54WJk1JSZGTJ0+av4f+HfwZ/2vCtyQkUDo0p2jgwEnK7bQjoFfAtSOLnEkNY4QHvkI7qloWVBeu0wACnhUWFmYmZevfxZ8RPMC3aOBA8IBcGEBwBRxAZvQqt3ZYU1NT5coVLjJ4Sp48eSQoKIiRH4IHAAAA76Yd1uDgYHMDPM2/x10AAAAAWEbwAAAAAMASggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlBA8AAAAALCF4AAAAAGAJwQMAAAAASwgeAAAAAFhC8AAAAADAEoIHAAAAAJYQPAAAAACwhOABAAAAgCUEDwAAAAAsIXgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAuS94eO211yQgIECGDBni2Hfx4kUZMGCAFCtWTPLnzy8dO3aUhIQEj7YTAAB4pxkzZkilSpUkb9680rBhQ4mLi7P0fV988YXpg7Rv397tbQS8mc8EDxs2bJB3331XatasmW7/0KFD5bvvvpMFCxbI6tWr5ejRo9KhQwePtRMAAHin+fPny7Bhw2T06NGyefNmqVWrlkRHR8uJEyeu+X1//vmnDB8+XO6+++4cayvgrXwieEhKSpInnnhC3n//fSlSpIhj/9mzZ2XOnDkyadIkadGihdSrV0/mzp0ra9eulXXr1nm0zQAAwLtof6FPnz7So0cPiYyMlFmzZklYWJh88MEHWX7PlStXTB/k5ZdflltuuSVH2wt4I58IHjQtqW3bttKqVat0+zdt2iSXL19Ot79GjRpSoUIFiY2NzfL1Ll26JOfOnXPcEhMT3dp+AADgPnoedz6v63k+o5SUFNNvcO4zBAYGmu1r9RnGjh0rJUuWlF69ermt/YAv8frgQXMMdWhxwoQJVz12/PhxCQkJkcKFC6fbHxERYR7Lir5WoUKFHDe9+gAAAHyTnsedz+uZ9RlOnTplRhG0j2C1z/Dzzz+bDAfNfADwtyDxYocPH5bBgwfL8uXLzcQmVxk5cqTJebSLj48ngAAAwEft3LlTypYt69gODQ11yWhG165dTeBQvHjxm349ILfw6uBBhxd1ElPdunUd+/SqwZo1a2T69OmybNkyMwx55syZdKMPWm2pVKlSWb6u/qfi/B+LDnECAADfVKBAASlYsOA1n6MBQJ48ea6qyJhVn2H//v1movRDDz3k2JeWlma+BgUFyZ49e6RKlSou+x0AX+HVaUstW7aUbdu2ydatWx23qKgoM3HJfj84OFhWrFjh+B79MB86dEgaNWrk0bYDAADvoWnOWljFuc+gwYBuZ9Zn0DmUGfsgDz/8sNx7773mfvny5XP4NwC8Q5C3X0m444470u0LDw83azrY9+sEJk1BKlq0qLnqMGjQIPOfwF133eWhVgMAAG+k/YXu3bubi48NGjSQKVOmSHJysqm+pLp162bSn3TOhKZLZ+yD2LMcMu4H/IlXBw9WTJ482VRL0MXhtLqC1mt+5513PN0sAADgZTp16iQnT56UmJgYM0m6du3asnTpUsckas1c0D4FgKwF2Gw2m/i5I0eOmOFHnaBdrlw5TzcHGSUni+TP//f9pCQdfvJ0iwD4qPMpqRIZs8zc3zk2WsJCfP4aml/j/A3kPMJrAAAAAJYQPAAAAACwhOABAAAAgCUEDwAAAAAsIXgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlBA8AAAAALCF4AAAAAGAJwQMAAAAASwgeAAAAAFhC8AAAAADAEoIHAAAAAJYQPAAAAACwhOABAAAAgCUEDwAAAAAsIXgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAlhA8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAADg+8HDzJkzpWbNmlKwYEFza9Sokfzwww+Oxy9evCgDBgyQYsWKSf78+aVjx46SkJDg0TYDAADvNWPGDKlUqZLkzZtXGjZsKHFxcVk+95tvvpGoqCgpXLiwhIeHS+3ateXjjz/O0fYC3sarg4dy5crJa6+9Jps2bZKNGzdKixYtpF27drJjxw7z+NChQ+W7776TBQsWyOrVq+Xo0aPSoUMHTzcbAAB4ofnz58uwYcNk9OjRsnnzZqlVq5ZER0fLiRMnMn1+0aJF5cUXX5TY2Fj57bffpEePHua2bNmyHG874C0CbDabTXyIfpDfeOMNefTRR6VEiRLy2Wefmftq9+7dctttt5kP+V133WX5NY8cOSLly5eXw4cPm4DFJfSwnj/vmtfyd8nJIhERf99PShIJD/d0iwD4qPMpqRIZ83fHb+NLrSQsJI+nm+Q38gXnkYCAAJe+ZnbP3zrSUL9+fZk+fbrZTktLM98/aNAgGTFihKWfWbduXWnbtq2MGzfuptsP+KIg8RFXrlwxIwzJyckmfUlHIy5fviytWrVyPKdGjRpSoUKF6wYPly5dMje7xMRE1zdYA4f8+V3/ugAAl4ga/x9PN8Gv7BwbLWEh7ul26Hn83Llzju3Q0FBzc5aSkmL6DiNHjnTsCwwMNP0I7Tdcj15r/emnn2TPnj3y+uuvu/g3AHyHV6ctqW3btpn5DPqfQL9+/eTbb7+VyMhIOX78uISEhJg8RGcRERHmsWuZMGGCFCpUyHHT14MPaNJEJCzM060A4ONXv6MqFvF0M+Bieh53Pq/reT6jU6dOmQuR2k/ITr/h7Nmzph+ifQ4dcXj77beldevWbvk9AF/g9SMP1atXl61bt5oP71dffSXdu3c38xtuhl510JxHu/j4eNcHENrJ1RQbuPaYunjIG4B/0bSZBf0ayYXLVzzdFL8M3Nxl586dUrZsWcd2xlGHm1GgQAHTD0lKSpIVK1aY/sMtt9wizZs3d9nPAHyJ1wcPGulXrVrV3K9Xr55s2LBBpk6dKp06dTJDkGfOnEk3+qDVlkqVKnXN18w4nOk81Oky2sklNx8AvDKAcFf6DDxDO/halfFaihcvLnny5LmqKuP1+g2a2mTvh2i1pV27dpmRDYIH+CuvT1vKSCc36XwFDSSCg4PNVQA7zUM8dOiQmRMBAADgfDFS+w7O/QbtU+h2dvoN9n4I4K+8+tKLphe1adPGTILWyVBaWWnVqlWmRJrmNPbq1csMH2oFJr3ioNUS9D+A7FRaAgAA/kH7DJr+rGs3NGjQQKZMmWIKsWj5VdWtWzeT/mSfM6Ff9blVqlQxAcOSJUvMOg+6DhXgr7w6eNC6y/pBPnbsmAkWdME4DRzsE5UmT55shhN1cTj9UGut5nfeecfTzQYAAF5IU55PnjwpMTExZpK0piEtXbrUMYlasxe0X2GngUX//v1NSdh8+fKZqo6ffPKJeR3AX/ncOg/u4JZ1HgAAgFtx/gZyns/NeQAAAADgGQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAAAAAsITgAQAAAIAlQdaelrulpaWZr8eOHfN0UwAAgEX287b9PA7A/QgeRCQhIcF8bdCggaebAgAAbuA8XqFCBU83A/ALATabzSZ+LjU1VbZs2SIRERESGOi6TK7ExESJjIyUnTt3SoECBVz2usgcxztncbxzFsc7Z3G8feN464iDBg516tSRoCCuhwI5geDBjc6dOyeFChWSs2fPSsGCBT3dnFyP452zON45i+OdszjeOYvjDfgOJkwDAAAAsITgAQAAAIAlBA9uFBoaKqNHjzZf4X4c75zF8c5ZHO+cxfHOWRxvwHcw5wEAAACAJYw8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcGDG82YMUMqVaokefPmlYYNG0pcXJynm5QrrFmzRh566CEpU6aMBAQEyMKFC9M9rjUAYmJipHTp0pIvXz5p1aqV7N2712Pt9WUTJkyQ+vXrmxVfS5YsKe3bt5c9e/ake87FixdlwIABUqxYMcmfP7907NjRrPiKGzNz5kypWbOmWShLb40aNZIffvjB8TjH231ee+0183/KkCFDHPs43q41ZswYc4ydbzVq1HA8zvEGvB/Bg5vMnz9fhg0bZkrPbd68WWrVqiXR0dFy4sQJTzfN5yUnJ5vjqcFZZiZOnCjTpk2TWbNmyfr16yU8PNwcez0pIXtWr15tTuTr1q2T5cuXy+XLl+W+++4zfwO7oUOHynfffScLFiwwzz969Kh06NDBo+32ZeXKlTOd2E2bNsnGjRulRYsW0q5dO9mxY4d5nOPtHhs2bJB3333XBG7OON6ud/vtt8uxY8cct59//tnxGMcb8AFaqhWu16BBA9uAAQMc21euXLGVKVPGNmHCBI+2K7fRt/C3337r2E5LS7OVKlXK9sYbbzj2nTlzxhYaGmr7/PPPPdTK3OPEiRPmmK9evdpxbIODg20LFixwPGfXrl3mObGxsR5sae5SpEgR2+zZsznebpKYmGirVq2abfny5bZmzZrZBg8ebPZzvF1v9OjRtlq1amX6GMcb8A2MPLhBSkqKuWqo6TJ2gYGBZjs2NtajbcvtDhw4IMePH0937AsVKmTSxjj2N+/s2bPma9GiRc1XfZ/raITz8dYUhAoVKnC8XeDKlSvyxRdfmJEeTV/ieLuHjq61bds23XFVHG/30DRSTTu95ZZb5IknnpBDhw6Z/RxvwDcEeboBudGpU6fMST8iIiLdft3evXu3x9rlDzRwUJkde/tjuDFpaWkmF7xJkyZyxx13mH16TENCQqRw4cLpnsvxvjnbtm0zwYKm2mne97fffiuRkZGydetWjreLaXCmqaWatpQR72/X0ws58+bNk+rVq5uUpZdfflnuvvtu2b59O8cb8BEEDwAsX53VE7xzfjLcQztWGijoSM9XX30l3bt3N/nfcK3Dhw/L4MGDzXweLWwB92vTpo3jvs4v0WCiYsWK8uWXX5oCFwC8H2lLblC8eHHJkyfPVRUidLtUqVIea5c/sB9fjr1rDRw4UL7//ntZuXKlmdBrp8dU0/TOnDmT7vkc75ujV1+rVq0q9erVMxWvtEDA1KlTOd4upmkyWsSibt26EhQUZG4apGnBBb2vV7w53u6lowy33nqr7Nu3j/c34CMIHtx04teT/ooVK9KlfOi2piLAfSpXrmxOMs7H/ty5c6bqEsc++3ROugYOmjbz008/mePrTN/nwcHB6Y63lnLVHGaOt+vo/x+XLl3ieLtYy5YtTYqYjvLYb1FRUSYP336f4+1eSUlJsn//flNam/c34BtIW3ITLdOqqQZ68mnQoIFMmTLFTHrs0aOHp5uWK042epXKeZK0nuh1Eq9OrNO8/PHjx0u1atVMZ3fUqFFmcp6uUYDspyp99tlnsmjRIrPWgz3vWCeha4qBfu3Vq5d5v+vx13UJBg0aZE70d911l6eb75NGjhxpUjv0vZyYmGiO/6pVq2TZsmUcbxfT97R9/o6dlnbWNQbs+znerjV8+HCzTo+mKmkZVi1nriP1Xbp04f0N+ApPl3vKzd5++21bhQoVbCEhIaZ067p16zzdpFxh5cqVpnRfxlv37t0d5VpHjRpli4iIMCVaW7ZsaduzZ4+nm+2TMjvOeps7d67jORcuXLD179/flBMNCwuzPfLII7Zjx455tN2+rGfPnraKFSua/zdKlChh3r8//vij43GOt3s5l2pVHG/X6tSpk6106dLm/V22bFmzvW/fPsfjHG/A+wXoP54OYAAAAAB4P+Y8AAAAALCE4AEAAACAJQQPAAAAACwheAAAAABgCcEDAAAAAEsIHgAAAABYQvAAAAAAwBKCBwAAAACWEDwAgAWrVq2SgIAAOXPmjKebAgCAx7DCNABk0Lx5c6ldu7ZMmTLFsS8lJUX++usviYiIMEEEAAD+iJEHAH7j8uXLN/y9ISEhUqpUKQIHAIBfI3gA4HKJiYnyxBNPSHh4uJQuXVomT55sruYPGTLEPH7p0iUZPny4lC1b1jynYcOGJi3Ibt68eVK4cGFZtmyZ3HbbbZI/f365//775dixY+l+zuzZs83jefPmlRo1asg777zjeOzPP/80Hf358+dLs2bNzHM+/fRT+d///iddunQxPzssLEzuvPNO+fzzzx3f99RTT8nq1atl6tSp5vv1pq+VWdrS119/LbfffruEhoZKpUqV5K233krXPt336quvSs+ePaVAgQJSoUIFee+999xyzAEAyAkEDwBcbtiwYfLLL7/Iv//9b1m+fLn897//lc2bNzseHzhwoMTGxsoXX3whv/32mzz22GMmONi7d6/jOefPn5c333xTPv74Y1mzZo0cOnTIBBx2GgjExMTIK6+8Irt27TKd9FGjRsmHH36Yri0jRoyQwYMHm+dER0fLxYsXpV69erJ48WLZvn279O3bV7p27SpxcXHm+Ro0NGrUSPr06WOCFb2VL1/+qt9x06ZN8o9//EM6d+4s27ZtkzFjxpifr4GPMw0ooqKiZMuWLdK/f395+umnZc+ePS493gAA5Bid8wAArnLu3DlbcHCwbcGCBY59Z86csYWFhdkGDx5sO3jwoC1Pnjy2+Pj4dN/XsmVL28iRI839uXPn6lws2759+xyPz5gxwxYREeHYrlKliu2zzz5L9xrjxo2zNWrUyNw/cOCAeY0pU6Zct81t27a1Pfvss47tZs2ambY6W7lypXm906dPm+3HH3/c1rp163TPee6552yRkZGO7YoVK9qefPJJx3ZaWpqtZMmStpkzZ163TQAAeKOgnAtTAPiDP/74w8wtaNCggWNfoUKFpHr16ua+XqW/cuWK3Hrrrem+T1OZihUr5tjWlKIqVao4tjX96cSJE+Z+cnKy7N+/X3r16mVGCOxSU1PNz3KmV/2d6c/WUYovv/xS4uPjzURo/dn687JDRzLatWuXbl+TJk3MJGv9GXny5DH7atas6Xhc05503oT99wAAwNcQPADIUUlJSaZjrWk/9g62nc5tsAsODk73mHa87cXh9DXU+++/b+ZLOMv4mjqnwtkbb7xhUpO0k6/zHfRxnYuhQYQ7ZPZ7pKWlueVnAQDgbgQPAFzqlltuMR3mDRs2mAnC6uzZs/L777/LPffcI3Xq1DFX5vXq+913331DP0PLpZYpU8aMcujE7OzQuRg6YvDkk0+abe3Ia9siIyPTVVbSNl6LTtTW18r42jqikjGAAQAgtyB4AOBSWlWoe/fu8txzz0nRokWlZMmSMnr0aAkMDDRX3bVzrR3+bt26mcnEGkycPHlSVqxYYVJ82rZta+nnvPzyy/LMM8+YNCWdbK2pRxs3bpTTp0+bCdtZqVatmnz11Veydu1aKVKkiEyaNEkSEhLSBQ9aJWn9+vWmypKOhujvkdGzzz4r9evXl3HjxkmnTp3MBPDp06enq/gEAEBuQ7UlAC6nHXKtWPTggw9Kq1atzFwAe0lVNXfuXBM8aAdc50K0b98+3UiFFb179zalWvW1NP1Iy7FqpaPKlStf8/teeuklqVu3rqm8pOVjdQ6C/nxnWtVJRw80oChRooSp9JSRvobOm9CKUXfccYep/DR27FhT6hUAgNyKFaYBuJ1OcNZ1FXSkQSc5AwAA30TaEgCX0zUNdu/ebSou6XwHvSKvMlYnAgAAvoXgAYBb6AJvuhiaTj7WRdl0objixYt7ulkAAOAmkLYEAAAAwBImTAMAAACwhOABAAAAgCUEDwAAAAAsIXgAAAAAYAnBAwAAAABLCB4AAAAAWELwAAAAAMASggcAAAAAYsX/AxXq5fhS3Md+AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tpot\n",
    "\n",
    "population_size=30\n",
    "initial_population_size=100\n",
    "population_scaling = .5\n",
    "generations_until_end_population = 50\n",
    "\n",
    "budget_range = [.3,1]\n",
    "generations_until_end_budget=50\n",
    "budget_scaling = .5\n",
    "stepwise_steps = 5\n",
    "\n",
    "#Population and budget use stepwise\n",
    "fig, ax1 = plt.subplots()\n",
    "ax2 = ax1.twinx()\n",
    "\n",
    "interpolated_values_population = tpot.utils.beta_interpolation(start=initial_population_size, end=population_size, n=generations_until_end_population, n_steps=stepwise_steps, scale=population_scaling)\n",
    "interpolated_values_budget = tpot.utils.beta_interpolation(start=budget_range[0], end=budget_range[1], n=generations_until_end_budget, n_steps=stepwise_steps, scale=budget_scaling)\n",
    "ax1.step(list(range(len(interpolated_values_population))), interpolated_values_population, label=f\"population size\")\n",
    "ax2.step(list(range(len(interpolated_values_budget))), interpolated_values_budget, label=f\"budget\", color='r')\n",
    "ax1.set_xlabel(\"generation\")\n",
    "ax1.set_ylabel(\"population size\")\n",
    "ax2.set_ylabel(\"bugdet\")\n",
    "\n",
    "ax1.legend(loc='center left', bbox_to_anchor=(1.1, 0.4))\n",
    "ax2.legend(loc='center left', bbox_to_anchor=(1.1, 0.3))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/metrics/_scorer.py:610: FutureWarning: The `needs_threshold` and `needs_proba` parameter are deprecated in version 1.4 and will be removed in 1.6. You can either let `response_method` be `None` or set it to `predict` to preserve the same behaviour.\n",
      "  warnings.warn(\n",
      "Generation:   2%|▏         | 1/50 [00:20<16:51, 20.64s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  1\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:   4%|▍         | 2/50 [00:45<18:39, 23.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  2\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:   6%|▌         | 3/50 [01:19<22:03, 28.17s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  3\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:   8%|▊         | 4/50 [01:57<24:30, 31.97s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  4\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  10%|█         | 5/50 [02:24<22:44, 30.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  5\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  12%|█▏        | 6/50 [03:09<25:40, 35.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  6\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  14%|█▍        | 7/50 [03:50<26:29, 36.96s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  7\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  16%|█▌        | 8/50 [04:27<26:04, 37.26s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  8\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  18%|█▊        | 9/50 [05:20<28:45, 42.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  9\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  20%|██        | 10/50 [06:03<28:09, 42.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  10\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  22%|██▏       | 11/50 [07:16<33:43, 51.88s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  11\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  24%|██▍       | 12/50 [08:04<31:55, 50.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  12\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  26%|██▌       | 13/50 [09:13<34:35, 56.10s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  13\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  28%|██▊       | 14/50 [10:15<34:49, 58.04s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  14\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  30%|███       | 15/50 [11:36<37:49, 64.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  15\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  32%|███▏      | 16/50 [12:59<39:47, 70.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  16\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  34%|███▍      | 17/50 [14:05<37:58, 69.05s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  17\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  36%|███▌      | 18/50 [15:23<38:13, 71.66s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  18\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  38%|███▊      | 19/50 [17:03<41:30, 80.33s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  19\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  40%|████      | 20/50 [18:32<41:28, 82.96s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  20\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  42%|████▏     | 21/50 [22:13<1:00:02, 124.23s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  21\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  44%|████▍     | 22/50 [24:54<1:03:11, 135.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  22\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  46%|████▌     | 23/50 [27:03<1:00:01, 133.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  23\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  48%|████▊     | 24/50 [29:09<56:48, 131.10s/it]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  24\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  50%|█████     | 25/50 [31:26<55:27, 133.09s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  25\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  52%|█████▏    | 26/50 [33:27<51:48, 129.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  26\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  54%|█████▍    | 27/50 [35:51<51:12, 133.60s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  27\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  56%|█████▌    | 28/50 [38:40<52:54, 144.28s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  28\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  58%|█████▊    | 29/50 [40:49<48:55, 139.80s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  29\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  60%|██████    | 30/50 [43:49<50:36, 151.83s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  30\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  62%|██████▏   | 31/50 [48:19<59:20, 187.37s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  31\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  64%|██████▍   | 32/50 [50:18<50:01, 166.77s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  32\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  66%|██████▌   | 33/50 [52:22<43:38, 154.01s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  33\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  68%|██████▊   | 34/50 [54:38<39:35, 148.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  34\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  70%|███████   | 35/50 [57:43<39:52, 159.52s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  35\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  72%|███████▏  | 36/50 [1:00:16<36:44, 157.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  36\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  74%|███████▍  | 37/50 [1:04:37<40:51, 188.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  37\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  76%|███████▌  | 38/50 [1:27:43<1:49:32, 547.68s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  38\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  78%|███████▊  | 39/50 [1:29:40<1:16:43, 418.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  39\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  80%|████████  | 40/50 [1:32:56<58:39, 351.95s/it]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  40\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  82%|████████▏ | 41/50 [1:36:55<47:41, 317.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  41\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  84%|████████▍ | 42/50 [1:38:54<34:27, 258.41s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  42\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  86%|████████▌ | 43/50 [1:40:59<25:28, 218.30s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  43\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  88%|████████▊ | 44/50 [1:42:38<18:14, 182.39s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  44\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  90%|█████████ | 45/50 [1:44:18<13:08, 157.68s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  45\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  92%|█████████▏| 46/50 [1:46:13<09:39, 144.91s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  46\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  94%|█████████▍| 47/50 [1:48:29<07:06, 142.24s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  47\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  96%|█████████▌| 48/50 [1:50:06<04:17, 128.67s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  48\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  98%|█████████▊| 49/50 [1:52:18<02:09, 129.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  49\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation: 100%|██████████| 50/50 [1:54:14<00:00, 137.09s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  50\n",
      "Best roc_auc_score score: 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total time: 6862.724096059799\n",
      "test score:  0.9917355371900827\n"
     ]
    }
   ],
   "source": [
    "# A Graph pipeline starting with at least one selector as a leaf, potentially followed by a series\n",
    "# of stacking classifiers or transformers, and ending with a classifier. The graph will have at most 15 nodes and a max depth of 6.\n",
    "\n",
    "import tpot\n",
    "import sklearn\n",
    "import sklearn.datasets\n",
    "import numpy as np\n",
    "import time\n",
    "import tpot\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import sklearn\n",
    "\n",
    "X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)\n",
    "scorer = sklearn.metrics.make_scorer(sklearn.metrics.roc_auc_score, needs_proba=True, multi_class='ovr')\n",
    "\n",
    "\n",
    "est = tpot.TPOTEstimator(\n",
    "    generations=50,\n",
    "    max_time_mins=None,\n",
    "    scorers=['roc_auc_ovr'],\n",
    "    scorers_weights=[1],\n",
    "    classification=True,\n",
    "    search_space = 'linear',\n",
    "    n_jobs=32,\n",
    "    cv=10,\n",
    "    verbose=3,\n",
    "\n",
    "    population_size=population_size,\n",
    "    initial_population_size=initial_population_size,\n",
    "    population_scaling = population_scaling,\n",
    "    generations_until_end_population = generations_until_end_population,\n",
    "    \n",
    "    budget_range = budget_range,\n",
    "    generations_until_end_budget=generations_until_end_budget,\n",
    "    )\n",
    "\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "est.fit(X_train, y_train)\n",
    "print(f\"total time: {time.time()-start}\")\n",
    "\n",
    "print(\"test score: \", scorer(est, X_test, y_test))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CV early pruning\n",
    "\n",
    "Most often, we will be evaluating pipelines using cross validation. However, we can often tell within the first few folds whether or not the pipeline is going have a reasonable change of outperforming the previous best pipelines. For example, if the best score so far is .92 AUROC and the average score of the first five folds of our current pipeline is only around .61, we can be reasonably confident that the next five folds are unlikely to this pipeline ahead of the others. We can save a significant amount of compute by not computing the rest of the folds. There are two strategies that TPOT can use to accomplish this (More information on these strategies in Tutorial 8).\n",
    "   1. Threshold Pruning: Pipelines must achieve a score above a predefined percentile threshold (based on previous pipeline scores) to proceed in each cross-validation (CV) fold.\n",
    "   2. Selection Pruning: Within each population, only the top N% of pipelines (ranked by performance in the previous CV fold) are selected to evaluate in the next fold.\"\n",
    "\n",
    "\n",
    "We can further reduce computational load by terminating the evaluation of individual pipelines early if the first few CV scores are not promising. Note that this is different than early stopping of the full algorithm. In this section we will cover:\n",
    "\n",
    "`threshold_evaluation_pruning`\n",
    "\n",
    "`threshold_evaluation_scaling`\n",
    "\n",
    "`min_history_threshold`\n",
    "\n",
    "`selection_evaluation_pruning`\n",
    "\n",
    "`selection_evaluation_scaling`\n",
    "\n",
    "Threshold early stopping uses previous scores to identify and terminate the cross validation evaluation of poorly performing pipelines. We calculate the percentile scores from the previously evaluated pipelines. A pipeline must reach the given percentile each fold for the next to be evaluated, otherwise the pipeline is discarded.\n",
    "\n",
    "The `threshold_evaluation_pruning` parameter is a list that specifies the starting and ending percentiles to use as a threshold for the evaluation early stopping. W The `threshold_evaluation_scaling` parameter is a float that controls the rate at which the threshold moves from the start to end percentile. The `min_history_threshold` parameter specifies the minimum number of previous scores needed before using threshold early stopping. This ensures that the algorithm has enough historical data to make an informed decision about when to stop evaluating pipelines.\n",
    "\n",
    "Selection early stopping uses a selection algorithm after each fold to select which algorithms will be evaluated for the next fold. For example, after evaluating 100 individuals on fold 1, we may want to only evaluate the best 50 for the remaining folds.\n",
    "\n",
    "The `selection_evaluation_pruning` parameter is a list that specifies the lower and upper percentage of the population size to select each round of CV. This is used to determine which individuals to evaluate in the next generation. The `selection_evaluation_scaling` parameter is a float that controls the rate at which the selection threshold moves from the start to end percentile.\n",
    "\n",
    "By manipulating these parameters, we can control how the algorithm selects individuals to evaluate in the next generation and when to stop evaluating pipelines that are not performing well.\n",
    "\n",
    "In practice, the values of these parameters will depend on the specific problem and the available computational resources. \n",
    "\n",
    "In the following sections, we will show you how to set and manipulate these parameters using Python code in a Jupyter Notebook. We will also provide examples of how these parameters can affect the performance of the algorithm.\n",
    "\n",
    "(Note that in these small test cases, you may not notice much or any performance improvements, these are more likely to be more beneficial in real world scenarios with larger datasets and slower evaluating pipelines.)\n",
    "\n",
    "**Considerations:**\n",
    "It is important to be aware of how CV pruning interacts with the evolutionary algorithm. When pipelines are pruned with one of these methods, they are removed from the live population and thus are no longer used to inform the TPOT algorithm. If too many pipelines are pruned, this could reduce the diversity of pipelines per generation, and limit TPOT's ability to learn. Additionally, the pruning methods may interact with how long it takes TPOT to run. If the pruning algorithm removes the slightly less performant but faster running pipelines, TPOT will most likely fill the next generation with only slower running pipelines, thus technically increasing the total runtime. This may be acceptable since more compute is dedicated to the higher performing pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAJPlJREFUeJzt3QtUVWXex/E/IDcvoGCCJHitkNQULEWtVsrEMI5pOpUtK0vLGTNSSU0mL3kLc0rNSkgXg120Jqers5aaUTlpYIjWWDamZWIiOJlAilxU3vU875zzelJLzwvs/Zzz/ay1l+fsfdj+7UDnx3P1qaurqxMAAAAD+VpdAAAAgLsIMgAAwFgEGQAAYCyCDAAAMBZBBgAAGIsgAwAAjEWQAQAAxmoiHu7MmTNSXFwsLVq0EB8fH6vLAQAAF0Etc/fTTz9JVFSU+Pr6em+QUSEmOjra6jIAAIAbDh48KO3atfPeIKNaYhz/IUJCQqwuBwAAXISKigrdEOH4HPfaIOPoTlIhhiADAIBZfm1YCIN9AQCAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYlgYZtavlpEmTpH379hIcHCz9+vWTgoICl50vZ82aJW3bttXXk5KSZO/evVaWDAAAbMTSIHP//ffLpk2b5OWXX5Zdu3bJzTffrMPKoUOH9PVFixbJsmXLJCsrS7Zt2ybNmjWT5ORkqaqqsrJsAABgEz51qtnDAidPntQ7Wr7zzjsyePBg5/mEhARJSUmRefPmSVRUlDzyyCMyZcoUfa28vFwiIiJk1apVMnLkyIvePTM0NFR/LZtGAgDsQH30nqw9LZ4i2N/vVzd3vFQX+/lt2e7Xp06dktOnT0tQUJDLedWFtGXLFtm/f7+UlJToFhoH9Q/q06eP5OXlXTDIVFdX6+Ps/xAAAM/gCQFANR/clpUnuw97zufT7rnJ0jTAmkhhWZBRrTGJiYm65aVr1666peXVV1/VIaVLly46xCjq/NnUc8e188nIyJA5c+Y0eP0AgMYPMX/IypPCA8esLgU2YlmQUdTYmDFjxsjll18ufn5+Eh8fL3feeacUFha6fc/09HRJS0tzaZGJjo6up4oBAFZRLTGeFGLi2obI2j8lSj33yFjWteSVQaZz586yefNmOXHihA4canbSHXfcIZ06dZLIyEj9mtLSUn3eQT3v2bPnBe8ZGBioDwCAZ3XJVNb8X/3bZyRJ0wDrPjztOq7EG1kaZBzUbCR1HDt2TDZu3KhnK3Xs2FGHmdzcXGdwUWFHzV4aP3681SUDgDE8sUtGhRirxmTAXiz9LlChRf2AXXXVVbJv3z6ZOnWqxMbGyn333adTqlpjZv78+XLFFVfoYDNz5kw9k2nYsGFWlg0ARvG0Lpne7VtZ2pUBe7E0yKgpVWpMy/fffy9hYWEyYsQIWbBggfj7++vr06ZN091O48aNk7KyMhkwYIBs2LDhnJlOAICLQ5cMPI1l68g0FtaRAeDtKmtOSdysjZZPkwUa4vObvZYAAICxCDIAAMBYBBkAAGAsOkoBwIvWXwE8DUEGALxo/RXA09C1BAAXwPorgP3RIgMAF4H1VwB7IsgAwEVgSXzAnuhaAgAAxiLIAAAAYxFkAACAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMxepOABoEmy0CaAwEGQD1js0WATQWupYA1Ds2WwTQWGiRAdCg2GwRQEMiyABoUGy2CKAh0bUEAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMFYTqwsA4Kqurk5O1p4Wk1XWmF0/AHMQZACbhZg/ZOVJ4YFjVpcCAEagawmwEdUS40khpnf7VhLs72d1GQA8GC0ygE1tn5EkTQPMDgEqxPj4+FhdBgAPRpABbEqFmKYB/IgCwC+hawkAABiLIAMAAIxFkAEAAMYiyAAAAGNZGmROnz4tM2fOlI4dO0pwcLB07txZ5s2bp9fScFCPZ82aJW3bttWvSUpKkr1791pZNgAAsAlLg8yTTz4pmZmZ8txzz8lXX32lny9atEieffZZ52vU82XLlklWVpZs27ZNmjVrJsnJyVJVVWVl6QAAwAYsndv5ySefyNChQ2Xw4MH6eYcOHeTVV1+VTz/91Nkas3TpUpkxY4Z+nfLSSy9JRESEvP322zJy5EgrywcAAN7cItOvXz/Jzc2Vr7/+Wj///PPPZcuWLZKSkqKf79+/X0pKSnR3kkNoaKj06dNH8vLyznvP6upqqaiocDkAAIBnsrRFZvr06TpoxMbGip+fnx4zs2DBAhk1apS+rkKMolpgzqaeO679XEZGhsyZM6cRqgcAAF7dIvP666/L6tWrZc2aNbJjxw558cUX5amnntJ/uis9PV3Ky8udx8GDB+u1ZgAAYB+WtshMnTpVt8o4xrp0795dDhw4oFtVRo8eLZGRkfp8aWmpnrXkoJ737NnzvPcMDAzUBwAA8HyWtshUVlaKr69rCaqL6cyZM/qxmpatwowaR+OguqLU7KXExMRGrxcAANiLpS0yQ4YM0WNiYmJi5Oqrr5adO3fK4sWLZcyYMfq62jV30qRJMn/+fLniiit0sFHrzkRFRcmwYcOsLB0AAHh7kFHrxahg8uCDD8qRI0d0QPnjH/+oF8BzmDZtmpw4cULGjRsnZWVlMmDAANmwYYMEBQVZWToAALABn7qzl9H1QKorSk3ZVgN/Q0JCrC4H+EWVNackbtZG/Xj33GRpGmDp7xoAYPvPb/ZaAgAAxiLIAAAAYxFkAACAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjsf45PIbabeNk7WkxWWWN2fUDQGMjyMBjQswfsvKk8MAxq0sBADQiupbgEVRLjCeFmN7tW0mwv5/VZQCA7dEiA4+zfUaSNA0wOwSoEOPj42N1GQBgewQZeBwVYpoG8K0NAN6AriUAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMBZBBgAAGIsgAwAAjEWQAQAAxiLIAAAAYxFkAACAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGNZGmQ6dOggPj4+5xwTJkzQ16uqqvTj8PBwad68uYwYMUJKS0utLBkAANiIpUGmoKBADh8+7Dw2bdqkz9922236z8mTJ8u6detk7dq1snnzZikuLpbhw4dbWTIAALCRJlb+5ZdddpnL84ULF0rnzp3lxhtvlPLycsnOzpY1a9bIwIED9fWcnBzp2rWr5OfnS9++fS2qGgAA2IVtxsjU1NTIK6+8ImPGjNHdS4WFhVJbWytJSUnO18TGxkpMTIzk5eVd8D7V1dVSUVHhcgAAAM9kmyDz9ttvS1lZmdx77736eUlJiQQEBEjLli1dXhcREaGvXUhGRoaEhoY6j+jo6AavHQAAeHmQUd1IKSkpEhUV9f+6T3p6uu6WchwHDx6stxoBAIC9WDpGxuHAgQPy/vvvy5tvvuk8FxkZqbubVCvN2a0yataSunYhgYGB+gAAAJ7PFi0yahBvmzZtZPDgwc5zCQkJ4u/vL7m5uc5ze/bskaKiIklMTLSoUgAAYCeWt8icOXNGB5nRo0dLkyb/V44a3zJ27FhJS0uTsLAwCQkJkdTUVB1imLFUv+rq6uRk7WkxWWWN2fUDAAwNMqpLSbWyqNlKP7dkyRLx9fXVC+Gp2UjJycmyfPlyS+r05BDzh6w8KTxwzOpSAAC4ZD516pPMg6np16p1Rw38Va06cFVZc0riZm0UT9G7fStZ+6dEPYUfAOD5n9+Wt8jAPrbPSJKmAX5ismB/P0IMAHgRggycVIhpGsC3BADAHLaYtQQAAOAOggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMBZBBgAAGIsgAwAAjEWQAQAAxiLIAAAAYxFkAACAsQgyAADAWAQZAADgfUHm448/lrvuuksSExPl0KFD+tzLL78sW7Zsqc/6AAAA6jfIvPHGG5KcnCzBwcGyc+dOqa6u1ufLy8vliSeecOeWAAAAjRNk5s+fL1lZWbJy5Urx9/d3nu/fv7/s2LHDnVsCAAA0TpDZs2eP3HDDDeecDw0NlbKyMnduCQAA0DhBJjIyUvbt23fOeTU+plOnTu7cEgAAoHGCzAMPPCATJ06Ubdu2iY+PjxQXF8vq1atlypQpMn78eHduCQAAcMmaXPqXiEyfPl3OnDkjgwYNksrKSt3NFBgYqINMamqqO7cEAABonCCjWmEee+wxmTp1qu5iOn78uMTFxUnz5s3duR0AAEDjBRmHgIAAHWAAAABsHWSGDx9+0Td988033a0HAACg/oOMmloNAABgZJDJyclp2EoAAAAuEZtGAgAAz2+RiY+Pl9zcXGnVqpX06tVLz1y6ELYpAAAAtgoyQ4cO1WvFOB7/UpABAACwVZCZPXu28/Hjjz/eUPUAAAA07BgZtZ/S0aNHzzmvNoxkryUAAGDrIPPdd9/J6dOnzzlfXV0t33//fX3UBQAAUL8r+7777rvOxxs3bnRZW0YFGzUYuGPHjpdySwAAgMYJMsOGDdN/qoG+o0ePdrnm7+8vHTp0kKefftr9agAAABoqyKgdrxXV6lJQUCCtW7e+lC8HAACwftPI/fv3128VAAAAjbn7tRoPo44jR444W2oc/vrXv7p7WwAAgIYNMnPmzJG5c+dK7969pW3btiyOBwAAzAkyWVlZsmrVKrn77rvrvyIAAICGXEempqZG+vXr586XAgAAWBtk7r//flmzZk29FHDo0CG56667JDw8XIKDg6V79+6yfft25/W6ujqZNWuW7sJS15OSkmTv3r318ncDAAAv7FqqqqqSFStWyPvvvy89evTQa8icbfHixRd1n2PHjkn//v3lpptukvXr18tll12mQ4raYdth0aJFsmzZMnnxxRf1tO+ZM2dKcnKy7N69W4KCgtwpHwAAeHOQ+de//iU9e/bUj7/44guXa5cy8PfJJ5+U6OhoycnJcZ47e2Vg1RqzdOlSmTFjht5xW3nppZckIiJC3n77bRk5cqQ75QMAAG8OMh9++GG9/OVqywPVunLbbbfJ5s2b5fLLL5cHH3xQHnjgAed6NSUlJbo7yUFti9CnTx/Jy8s7b5BR+z2pw6GioqJeagUAAB4yRsZh3759es+lkydPOltQLsW3334rmZmZcsUVV+j7jB8/Xh5++GHdjaSoEKOoFpizqeeOaz+XkZGhw47jUC0+AADAM7kVZI4ePSqDBg2SK6+8Un73u9/J4cOH9fmxY8fKI488ctH3UQvpxcfHyxNPPCG9evWScePG6dYYNb3bXenp6VJeXu48Dh486Pa9AACABwaZyZMn6wG+RUVF0rRpU+f5O+64QzZs2HDR91EzkeLi4lzOde3aVd9XiYyM1H+Wlpa6vEY9d1z7ucDAQAkJCXE5AACAZ3IryLz33nt6oG67du1czqsuogMHDlz0fdSMpT179ric+/rrr6V9+/bOgb8qsKitEM4e87Jt2zZJTEx0p3QAAODtg31PnDjh0hLj8OOPP+oWkUtp2VEL66mupdtvv10+/fRTPa1bHY4ZUJMmTZL58+frkOSYfh0VFSXDhg1zp3QAAODtLTLXX3+9ngbtoAKHGu+i1nxRa8JcrGuvvVbeeustefXVV6Vbt24yb948Pd161KhRztdMmzZNUlNT9fgZ9frjx4/r7ivWkAEAAD51lzrV6L9rx6jBvmqg7gcffCC33HKLfPnll7pFZuvWrdK5c2exC9UVpWYvqYG/jJc5V2XNKYmbtVE/3j03WZoGuL0hOgAAjf757VaLjGo9UWNZBgwYoBeqU11Nw4cPl507d9oqxAAAAM/m9q/fKiU99thj9VsNAADAJXCrRUZtKbB27dpzzqtzjsXsAAAAbBlk1Oq5rVu3Pud8mzZt9AwkAAAA2wYZtWDd2Zs7Oqj1XxyL2QEAANgyyKiWF7UD9s99/vnnEh4eXh91AQAANMxg3zvvvFNv7tiiRQu54YYb9Dm1e/XEiRPPuyO1J1Kz1k/WnhbTVdaY/28AAHgvt4KMWrjuu+++02vJNGnyv7dQC+Ldc889XjFGRoWYP2TlSeGBY1aXAgCAV2vizod4SUmJrFq1Sm8d8Nlnn0lwcLB0797duUeSp1MtMZ4WYnq3byXB/n5WlwEAQMMHmS5duuiVfNX+R+rwZttnJEnTAPMDgAoxaqsJAAA8Osj4+vrq8HL06FGvDzGKCjEs6w8AgEGzlhYuXChTp07Vey4BAABYxa2mBDWot7KyUq655hoJCAjQY2TOpjaPBAAAsGWQWbp0af1XAgAA0BhBZvTo0e58GQAAgPVjZJRvvvlGZsyYoRfHO3LkiD63fv16PZsJAADAtkFGreKr1o3Ztm2bvPnmm3L8+HHnFgWzZ8+u7xoBAADqL8hMnz5dL4a3adMmPdjXYeDAgZKfn+/OLQEAABonyOzatUtuvfXW824m+cMPP7hzSwAAgMYJMi1btpTDhw+fc37nzp1y+eWXu3NLAACAxgkyaofrRx99VO+5pJa1VxtGbt26VaZMmaLXmAEAALBtkFE7XMfGxkp0dLQe6BsXFyfXX3+99OvXT89kAgAAsO06MmqA78qVK2XWrFl6vMyJEyekV69eejNJAACAxuL2bofZ2dmyZMkS2bt3r36uNpCcNGmS3H///fVZHwAAQP0GGdUSs3jxYklNTZXExER9Li8vTyZPnixFRUUyd+5cd24LAADQ8EEmMzNTdy2pVX0dbrnlFunRo4cONwQZAABg28G+tbW10rt373POJyQkyKlTp+qjLgAAgIYJMnfffbdulfm5FStWyKhRo9y5JQAAQOMO9n3vvfekb9+++rnad0mNj1HryKSlpTlfp8bSAAAA2CbIfPHFFxIfH+/cBVtp3bq1PtQ1B7VYHgAAgK2CzIcfflj/lQAAADTGGBkAAAA7IMgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMJalQebxxx8XHx8flyM2NtZ5vaqqSiZMmCDh4eHSvHlzGTFihJSWllpZMgAAsBHLW2SuvvpqOXz4sPPYsmWL89rkyZNl3bp1snbtWtm8ebMUFxfL8OHDLa0XAADYRxPLC2jSRCIjI885X15eLtnZ2bJmzRoZOHCgPpeTkyNdu3aV/Px86du3rwXVAgAAO7G8RWbv3r0SFRUlnTp1klGjRklRUZE+X1hYKLW1tZKUlOR8rep2iomJkby8vAver7q6WioqKlwOAADgmSwNMn369JFVq1bJhg0bJDMzU/bv3y/XX3+9/PTTT1JSUiIBAQHSsmVLl6+JiIjQ1y4kIyNDQkNDnUd0dHQj/EsAAIDXdS2lpKQ4H/fo0UMHm/bt28vrr78uwcHBbt0zPT1d0tLSnM9ViwxhBgAAz2R519LZVOvLlVdeKfv27dPjZmpqaqSsrMzlNWrW0vnG1DgEBgZKSEiIywEAADyTrYLM8ePH5ZtvvpG2bdtKQkKC+Pv7S25urvP6nj179BiaxMRES+sEAAD2YGnX0pQpU2TIkCG6O0lNrZ49e7b4+fnJnXfeqce3jB07VncThYWF6ZaV1NRUHWKYsQQAACwPMt9//70OLUePHpXLLrtMBgwYoKdWq8fKkiVLxNfXVy+Ep2YjJScny/Lly3nnAACA5lNXV1cnHkwN9lWtO2pdmvoaL1NZc0riZm3Uj3fPTZamAZYvxwMAgFd+fttqjAwAAMClIMgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMBZBBgAAGIsgAwAAjEWQAQAAxiLIAAAAYxFkAACAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMBZBBgAAGIsgAwAAjEWQAQAAxiLIAAAAYxFkAACAsQgyAADAWAQZAABgLNsEmYULF4qPj49MmjTJea6qqkomTJgg4eHh0rx5cxkxYoSUlpZaWicAALAPWwSZgoICeeGFF6RHjx4u5ydPnizr1q2TtWvXyubNm6W4uFiGDx9uWZ0AAMBeLA8yx48fl1GjRsnKlSulVatWzvPl5eWSnZ0tixcvloEDB0pCQoLk5OTIJ598Ivn5+ZbWDAAA7MHyIKO6jgYPHixJSUku5wsLC6W2ttblfGxsrMTExEheXt4F71ddXS0VFRUuBwAA8ExNrPzLX3vtNdmxY4fuWvq5kpISCQgIkJYtW7qcj4iI0NcuJCMjQ+bMmdMg9QIAAHuxrEXm4MGDMnHiRFm9erUEBQXV233T09N1t5TjUH8PAADwTJYFGdV1dOTIEYmPj5cmTZroQw3oXbZsmX6sWl5qamqkrKzM5evUrKXIyMgL3jcwMFBCQkJcDgAA4Jks61oaNGiQ7Nq1y+Xcfffdp8fBPProoxIdHS3+/v6Sm5urp10re/bskaKiIklMTLSoagAAYCeWBZkWLVpIt27dXM41a9ZMrxnjOD927FhJS0uTsLAw3bKSmpqqQ0zfvn0tqhoAANiJpYN9f82SJUvE19dXt8io2UjJycmyfPlyq8sCAAA2Yasg89FHH7k8V4OAn3/+eX0AAADYbh0ZAAAAdxFkAACAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMBZBBgAAGIsgAwAAjEWQAQAAxiLIAAAAYxFkAACAsQgyAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDAAAMJalQSYzM1N69OghISEh+khMTJT169c7r1dVVcmECRMkPDxcmjdvLiNGjJDS0lIrSwYAADZiaZBp166dLFy4UAoLC2X79u0ycOBAGTp0qHz55Zf6+uTJk2XdunWydu1a2bx5sxQXF8vw4cOtLBkAANhIEyv/8iFDhrg8X7BggW6lyc/P1yEnOztb1qxZowOOkpOTI127dtXX+/bta1HVAADALmwzRub06dPy2muvyYkTJ3QXk2qlqa2tlaSkJOdrYmNjJSYmRvLy8i54n+rqaqmoqHA5AACAZ7I8yOzatUuPfwkMDJQ//elP8tZbb0lcXJyUlJRIQECAtGzZ0uX1ERER+tqFZGRkSGhoqPOIjo5uhH8FAADwuq4l5aqrrpLPPvtMysvL5e9//7uMHj1aj4dxV3p6uqSlpTmfqxaZ+g4zwf5+sntusvMxAADw0iCjWl26dOmiHyckJEhBQYE888wzcscdd0hNTY2UlZW5tMqoWUuRkZEXvJ9q2VFHQ/Lx8ZGmAZb/pwMAwOtZ3rX0c2fOnNHjXFSo8ff3l9zcXOe1PXv2SFFRkR5DAwAAYGmzguoGSklJ0QN4f/rpJz1D6aOPPpKNGzfq8S1jx47V3URhYWF6nZnU1FQdYpixBAAALA8yR44ckXvuuUcOHz6sg4taHE+FmN/85jf6+pIlS8TX11cvhKdaaZKTk2X58uW8cwAAQPOpq6urEw+mBvuqkKQGE6tWHQAA4Dmf37YbIwMAAHCxCDIAAMBYBBkAAGAsggwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLE8fgtnx8LFaoVAAABgBsfn9q9tQODxQUZtRqlER0dbXQoAAHDjc1xtVeC1ey2dOXNGiouLpUWLFuLj41OvSVGFo4MHD7KHk03wntgL74e98H7YC+/Hr1PxRIWYqKgovYG017bIqH98u3btGuz+6huQb0J74T2xF94Pe+H9sBfej1/2Sy0xDgz2BQAAxiLIAAAAYxFk3BQYGCizZ8/Wf8IeeE/shffDXng/7IX3o/54/GBfAADguWiRAQAAxiLIAAAAYxFkAACAsQgyAADAWAQZNz3//PPSoUMHCQoKkj59+sinn35qdUleKSMjQ6699lq9cnObNm1k2LBhsmfPHqvLwn8tXLhQr6g9adIkq0vxaocOHZK77rpLwsPDJTg4WLp37y7bt2+3uiyvdPr0aZk5c6Z07NhRvxedO3eWefPm/ep+Qrgwgowb/va3v0laWpqeOrdjxw655pprJDk5WY4cOWJ1aV5n8+bNMmHCBMnPz5dNmzZJbW2t3HzzzXLixAmrS/N6BQUF8sILL0iPHj2sLsWrHTt2TPr37y/+/v6yfv162b17tzz99NPSqlUrq0vzSk8++aRkZmbKc889J1999ZV+vmjRInn22WetLs1YTL92g2qBUa0A6hvRsZ+T2jMjNTVVpk+fbnV5Xu0///mPbplRAeeGG26wuhyvdfz4cYmPj5fly5fL/PnzpWfPnrJ06VKry/JK6v9JW7dulY8//tjqUiAiv//97yUiIkKys7Od50aMGKFbZ1555RVLazMVLTKXqKamRgoLCyUpKcllPyf1PC8vz9LaIFJeXq7/DAsLs7oUr6ZayQYPHuzycwJrvPvuu9K7d2+57bbbdMjv1auXrFy50uqyvFa/fv0kNzdXvv76a/38888/ly1btkhKSorVpRnL4zeNrG8//PCD7uNUifps6vm///1vy+rC/7aMqbEYqhm9W7duVpfjtV577TXd5aq6lmC9b7/9VndlqO7wP//5z/p9efjhhyUgIEBGjx5tdXle2UKmdr6OjY0VPz8//XmyYMECGTVqlNWlGYsgA49qBfjiiy/0bzewxsGDB2XixIl6vJIaCA97BHzVIvPEE0/o56pFRv2cZGVlEWQs8Prrr8vq1atlzZo1cvXVV8tnn32mfwGLiori/XATQeYStW7dWqfo0tJSl/PqeWRkpGV1ebuHHnpI/vGPf8g///lPadeundXleC3V7aoGvavxMQ7qN071vqgxZdXV1frnB42nbdu2EhcX53Kua9eu8sYbb1hWkzebOnWqbpUZOXKkfq5mkB04cEDPwCTIuIcxMpdINccmJCToPs6zf+NRzxMTEy2tzRupseoqxLz11lvywQcf6CmNsM6gQYNk165d+rdMx6FaA1SzuXpMiGl8qqv150sSqPEZ7du3t6wmb1ZZWanHVZ5N/VyozxG4hxYZN6i+ZpWc1f+gr7vuOj0bQ033ve+++6wuzSu7k1QT7TvvvKPXkikpKdHnQ0ND9SwANC71Hvx8fFKzZs30+iWMW7LG5MmT9QBT1bV0++236zWvVqxYoQ80viFDhugxMTExMbpraefOnbJ48WIZM2aM1aWZS02/xqV79tln62JiYuoCAgLqrrvuurr8/HyrS/JK6lv4fEdOTo7VpeG/brzxxrqJEydaXYZXW7duXV23bt3qAgMD62JjY+tWrFhhdUleq6KiQv88qM+PoKCguk6dOtU99thjddXV1VaXZizWkQEAAMZijAwAADAWQQYAABiLIAMAAIxFkAEAAMYiyAAAAGMRZAAAgLEIMgAAwFgEGQAAYCyCDABbUWt0jhs3TsLCwsTHx0fv0fRLPvroI/26srKyC75m1apV0rJlywaoFoDV2GsJgK1s2LBBBw8VUDp16qR3nAeACyHIALCVb775Rtq2bas3OgSAX0PXEgDbuPfeeyU1NVWKiop0d1GHDh2kurpaHn74YWnTpo0EBQXJgAEDpKCg4Bfvo1p01O7CTZs2lVtvvVWOHj3aaP8GAI2LIAPANp555hmZO3eutGvXTg4fPqwDy7Rp0+SNN96QF198UXbs2CFdunSR5ORk+fHHH897j23btsnYsWPloYce0uNrbrrpJpk/f36j/1sANA6CDADbCA0NlRYtWoifn59ERkbqFpXMzEz5y1/+IikpKRIXFycrV66U4OBgyc7OvmAY+u1vf6sD0JVXXqlbc1TwAeCZCDIAbD1epra2Vvr37+885+/vL9ddd5189dVX5/0adb5Pnz4u5xITExu8VgDWIMgAAABjEWQA2Fbnzp0lICBAtm7d6jynWmjU2BnVzXQ+Xbt21eNkzpafn9/gtQKwBtOvAdhWs2bNZPz48TJ16lS9QJ6aibRo0SKprKzUA3rPR42JUV1RTz31lAwdOlQ2btyo16YB4JlokQFgawsXLpQRI0bI3XffLfHx8bJv3z4dTlq1anXe1/ft21cPCFaDfq+55hp57733ZMaMGY1eN4DG4VOn1gMHAAAwEC0yAADAWAQZAABgLIIMAAAwFkEGAAAYiyADAACMRZABAADGIsgAAABjEWQAAICxCDIAAMBYBBkAAGAsggwAABBT/Q9yZSFTz9dH2gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tpot\n",
    "import time\n",
    "import sklearn\n",
    "import sklearn.datasets\n",
    "\n",
    "threshold_evaluation_pruning = [30, 90]\n",
    "threshold_evaluation_scaling = .2 #.5\n",
    "cv = 10\n",
    "\n",
    "#Population and budget use stepwise\n",
    "fig, ax1 = plt.subplots()\n",
    "\n",
    "interpolated_values = tpot.utils.beta_interpolation(start=threshold_evaluation_pruning[0], end=threshold_evaluation_pruning[-1], n=cv, n_steps=cv, scale=threshold_evaluation_scaling)\n",
    "ax1.step(list(range(len(interpolated_values))), interpolated_values, label=f\"threshold\")\n",
    "ax1.set_xlabel(\"fold\")\n",
    "ax1.set_ylabel(\"percentile\")\n",
    "#ax1.legend(loc='center left', bbox_to_anchor=(1.1, 0.4))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tpot\n",
    "from tpot.search_spaces.pipelines import *\n",
    "from tpot.search_spaces.nodes import *\n",
    "from tpot.config.get_configspace import get_search_space\n",
    "import sklearn.model_selection\n",
    "import sklearn\n",
    "\n",
    "\n",
    "selectors = get_search_space([\"selectors\",\"selectors_classification\", \"Passthrough\"], random_state=42,)\n",
    "estimators = get_search_space(['XGBClassifier'],random_state=42,)\n",
    "\n",
    "scalers = get_search_space([\"scalers\",\"Passthrough\"],random_state=42,)\n",
    "\n",
    "transformers_layer =UnionPipeline([\n",
    "                        ChoicePipeline([\n",
    "                            DynamicUnionPipeline(get_search_space([\"transformers\"], random_state=42,)),\n",
    "                            get_search_space(\"SkipTransformer\"),\n",
    "                        ]),\n",
    "                        get_search_space(\"Passthrough\")\n",
    "                        ]\n",
    "                    )\n",
    "    \n",
    "search_space = SequentialPipeline(search_spaces=[\n",
    "                                            scalers,\n",
    "                                            selectors, \n",
    "                                            transformers_layer,\n",
    "                                            estimators,\n",
    "                                            ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/metrics/_scorer.py:610: FutureWarning: The `needs_threshold` and `needs_proba` parameter are deprecated in version 1.4 and will be removed in 1.6. You can either let `response_method` be `None` or set it to `predict` to preserve the same behaviour.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tpot\n",
    "import time\n",
    "import sklearn\n",
    "import sklearn.datasets\n",
    "\n",
    "scorer = sklearn.metrics.make_scorer(sklearn.metrics.roc_auc_score, needs_proba=True, multi_class='ovr')\n",
    "\n",
    "X, y = sklearn.datasets.make_classification(n_samples=5000, n_features=20, n_classes=5, random_state=1, n_informative=15, n_redundant=5, n_repeated=0, n_clusters_per_class=3, class_sep=.8)\n",
    "X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)\n",
    "\n",
    "# search_space = tpot.config.template_search_spaces.get_template_search_spaces(\"linear\",inner_predictors=False, random_state=42)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  10%|█         | 1/10 [02:42<24:26, 162.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  1\n",
      "Best roc_auc_score score: 0.9212394545585599\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  20%|██        | 2/10 [06:10<25:14, 189.31s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  2\n",
      "Best roc_auc_score score: 0.921316057689257\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  30%|███       | 3/10 [10:07<24:37, 211.00s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  3\n",
      "Best roc_auc_score score: 0.9291812014325632\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  40%|████      | 4/10 [16:26<27:43, 277.33s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  4\n",
      "Best roc_auc_score score: 0.9291812014325632\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  50%|█████     | 5/10 [21:24<23:44, 284.90s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  5\n",
      "Best roc_auc_score score: 0.9309353469187138\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  60%|██████    | 6/10 [28:02<21:32, 323.19s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  6\n",
      "Best roc_auc_score score: 0.9328394699598583\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  70%|███████   | 7/10 [36:02<18:43, 374.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  7\n",
      "Best roc_auc_score score: 0.9341963775600117\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  80%|████████  | 8/10 [45:34<14:34, 437.41s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  8\n",
      "Best roc_auc_score score: 0.9341963775600117\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  90%|█████████ | 9/10 [54:40<07:51, 471.27s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  9\n",
      "Best roc_auc_score score: 0.9356175936945494\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation: 100%|██████████| 10/10 [1:03:45<00:00, 382.55s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  10\n",
      "Best roc_auc_score score: 0.9371852416832148\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total time: 3836.4180731773376\n",
      "test score:  0.9422368174356803\n"
     ]
    }
   ],
   "source": [
    "# no pruning\n",
    "est = tpot.TPOTEstimator(  \n",
    "                            generations=10,\n",
    "                            max_time_mins=None,\n",
    "                            scorers=['roc_auc_ovr'],\n",
    "                            scorers_weights=[1],\n",
    "                            classification=True,\n",
    "                            search_space = search_space,\n",
    "                            population_size=100,\n",
    "                            n_jobs=32,\n",
    "                            cv=cv,\n",
    "                            verbose=3,\n",
    "                            random_state=42,\n",
    "                            )\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "est.fit(X_train, y_train)\n",
    "print(f\"total time: {time.time()-start}\")\n",
    "print(\"test score: \", scorer(est, X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  10%|█         | 1/10 [02:57<26:40, 177.87s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  1\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  20%|██        | 2/10 [03:57<14:24, 108.05s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  2\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  30%|███       | 3/10 [05:58<13:18, 114.13s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  3\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  40%|████      | 4/10 [07:54<11:29, 114.96s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  4\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  50%|█████     | 5/10 [10:43<11:11, 134.34s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  5\n",
      "Best roc_auc_score score: 0.921316057689257\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  60%|██████    | 6/10 [13:16<09:23, 140.78s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  6\n",
      "Best roc_auc_score score: 0.921316057689257\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  70%|███████   | 7/10 [15:05<06:31, 130.43s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  7\n",
      "Best roc_auc_score score: 0.921316057689257\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  80%|████████  | 8/10 [18:01<04:49, 144.72s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  8\n",
      "Best roc_auc_score score: 0.9255953925256337\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  90%|█████████ | 9/10 [19:53<02:14, 134.59s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  9\n",
      "Best roc_auc_score score: 0.9255953925256337\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation: 100%|██████████| 10/10 [21:24<00:00, 128.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  10\n",
      "Best roc_auc_score score: 0.9255953925256337\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total time: 1295.825649023056\n",
      "test score:  0.9320499022897322\n"
     ]
    }
   ],
   "source": [
    "import tpot.config\n",
    "import tpot.config.template_search_spaces\n",
    "import tpot.search_spaces\n",
    "\n",
    "\n",
    "\n",
    "# search_space = tpot.config.get_search_space([\"RandomForestClassifier\"])\n",
    "\n",
    "est = tpot.TPOTEstimator(  \n",
    "                            generations=10,\n",
    "                            max_time_mins=None,\n",
    "                            scorers=['roc_auc_ovr'],\n",
    "                            scorers_weights=[1],\n",
    "                            classification=True,\n",
    "                            search_space = search_space,\n",
    "                            population_size=100,\n",
    "                            n_jobs=32,\n",
    "                            cv=cv,\n",
    "                            verbose=3,\n",
    "                            random_state=42,\n",
    "\n",
    "                            threshold_evaluation_pruning = threshold_evaluation_pruning,\n",
    "                            threshold_evaluation_scaling = threshold_evaluation_scaling,\n",
    "                            )\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "est.fit(X_train, y_train)\n",
    "print(f\"total time: {time.time()-start}\")\n",
    "print(\"test score: \", scorer(est, X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKVpJREFUeJzt3QtUlVXex/E/oNwcQQ0Vb4lipuQFlSC8zMq0mGxKrWk0TRlTW5WXktIkFRNNvIwuy0zSNCvHybGcbHVBTbNJI0nU1LyllpiJeMkbKCjyrr3nhREB4+DhPOfs8/2s9SzOs3meczYe9fzYV4+CgoICAQAAMISn1RUAAACwJ8INAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRqoibuXr1qvz6669SvXp18fDwsLo6AACgHNSyfOfPn5f69euLp+eN22bcLtyoYNOoUSOrqwEAACrgyJEj0rBhwxte43bhRrXYFP7hBAQEWF0dAABQDufOndONE4Wf4zfiduGmsCtKBRvCDQAArqU8Q0oYUAwAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARrE83MybN09CQkLE19dXoqKiJC0trcxrL1++LImJiRIaGqqvb9u2raSkpDi0vgAAwLlZGm6WL18ucXFxMnHiRNm6dasOKzExMZKVlVXq9ePHj5c333xT5s6dK7t375annnpKevfuLdu2bXN43QEAgHPyKCgoKLDqxVVLzZ133imvv/66Pr969are8XPEiBEyduzYEtfXr19fxo0bJ8OGDSsqe+SRR8TPz0+WLl1a7l1FAwMD5ezZs3bdOFP9MV68nK8f+1X1KtfGXgAAQOz++W1Zy01eXp6kp6dL9+7d/1cZT099npqaWuo9ubm5ujvqWirYbNy4sczXUfeoP5Brj8qggk1Ywmp9FIYcAADgeJaFm5MnT0p+fr7UrVu3WLk6z8zMLPUe1WU1e/Zs+fHHH3Urz9q1a2XlypVy7NixMl8nKSlJJ73CQ7UMAQAAc1k+oNgWr776qtx2223SokUL8fb2luHDh8ugQYN0i09Z4uPjdRNW4XHkyBGH1hkAALhJuAkKChIvLy85fvx4sXJ1HhwcXOo9tWvXlo8++kiys7Pl8OHDsnfvXvnDH/4gTZs2LfN1fHx8dN/ctQcAADCXZeFGtbx06NBB1q1bV1SmuprUeXR09A3vVeNuGjRoIFeuXJEPP/xQevbs6YAaAwAAV1DFyhdX08BjY2MlIiJCIiMjZc6cObpVRnU1KQMHDtQhRo2bUTZv3ixHjx6V8PBw/fXll1/WgWjMmDFW/hgAAMCJWBpu+vTpIydOnJCEhAQ9iFiFFrUoX+Eg44yMjGLjaS5duqTXujl06JDujurRo4e89957UqNGDQt/CgAA4EwsXefGCpW1zk1O3hU9DVzZnRgj/t6W5kYAAIziEuvcAAAAVAbCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEaxPNzMmzdPQkJCxNfXV6KioiQtLe2G18+ZM0duv/128fPzk0aNGsmoUaPk0qVLDqsvAABwbpaGm+XLl0tcXJxMnDhRtm7dKm3btpWYmBjJysoq9fply5bJ2LFj9fV79uyRRYsW6ed46aWXHF53AADgnCwNN7Nnz5ahQ4fKoEGDJCwsTJKTk8Xf318WL15c6vXffPONdOrUSfr166dbe+677z557LHHfre1BwAAuA/Lwk1eXp6kp6dL9+7d/1cZT099npqaWuo9HTt21PcUhplDhw7JZ599Jj169CjzdXJzc+XcuXPFDgAAYK4qVr3wyZMnJT8/X+rWrVusXJ3v3bu31HtUi426r3PnzlJQUCBXrlyRp5566obdUklJSTJp0iS71x8AADgnywcU22LDhg0ydepUeeONN/QYnZUrV8qnn34qkydPLvOe+Ph4OXv2bNFx5MgRh9YZAAC4SctNUFCQeHl5yfHjx4uVq/Pg4OBS75kwYYIMGDBAhgwZos9bt24t2dnZ8uSTT8q4ceN0t9b1fHx89AEAANyDZS033t7e0qFDB1m3bl1R2dWrV/V5dHR0qffk5OSUCDAqICmqmwoAAMCylhtFTQOPjY2ViIgIiYyM1GvYqJYYNXtKGThwoDRo0ECPm1EefPBBPcOqXbt2ek2cAwcO6NYcVV4YcgAAgHuzNNz06dNHTpw4IQkJCZKZmSnh4eGSkpJSNMg4IyOjWEvN+PHjxcPDQ389evSo1K5dWwebV155xcKfAgAAOBOPAjfrz1FTwQMDA/Xg4oCAALs9b07eFQlLWK0f706MEX9vS3MjAABu+/ntUrOlAAAAfg/hBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKM4RbiZN2+ehISEiK+vr0RFRUlaWlqZ1959993i4eFR4njggQccWmcAAOCcLA83y5cvl7i4OJk4caJs3bpV2rZtKzExMZKVlVXq9StXrpRjx44VHbt27RIvLy959NFHHV53AADgfCwPN7Nnz5ahQ4fKoEGDJCwsTJKTk8Xf318WL15c6vW1atWS4ODgomPt2rX6esINAACoULi555575MyZMyXKz507p79ni7y8PElPT5fu3bsXlXl6eurz1NTUcj3HokWLpG/fvlKtWrVSv5+bm6vrdu0BAADMZXO42bBhgw4l17t06ZJ8/fXXNj3XyZMnJT8/X+rWrVusXJ1nZmb+7v1qbI7qlhoyZEiZ1yQlJUlgYGDR0ahRI5vqCAAAXEuV8l64Y8eOose7d+8uFj5UQElJSZEGDRqII6lWm9atW0tkZGSZ18THx+sxPYVUyw0BBwAAc5U73ISHhxfNTCqt+8nPz0/mzp1r04sHBQXpwcDHjx8vVq7O1XiaG8nOzpb3339fEhMTb3idj4+PPgAAgHsod7j56aefpKCgQJo2baq7g2rXrl30PW9vb6lTp44OKrZQ93Xo0EHWrVsnvXr10mVXr17V58OHD7/hvStWrNDjaR5//HGbXhMAAJit3OGmcePGReHDnlSXUWxsrEREROjupTlz5uhWGTV7Shk4cKDu7lJjZ67vklKB6JZbbrFrfQAAgJuEm0IqZKgBv0888USxcjV1+8SJE/Liiy/a9Hx9+vTR9yUkJOhxPKr7S43fKRxknJGRoWdQXWvfvn2yceNGWbNmja3VBwAAhvMoUH1NNlArCS9btkw6duxYrHzz5s16SrbqvnJmakCxmjV19uxZCQgIsNvz5uRdkbCE1frx7sQY8fe2OTcCAAA7fH7bPBVcta7Uq1evRLkag6NWDAYAALCSzeFGTaPetGlTiXJVVr9+fXvVCwAAoEJs7jtRWyU899xzcvny5aIp4Wp205gxY+T555+vWC0AAACsCjejR4+WU6dOyTPPPFO0UrHazVsNJFYL5gEAALhUuFGL+E2fPl0mTJgge/bs0Yv33XbbbSyUBwAAXHtXcDWw+PTp0xIaGqqDjY2TrgAAAJwj3KguqW7duknz5s2lR48eRTOkBg8ezJgbAADgeuFm1KhRUrVqVb24nr+/f7HF+NTiewAAAC415katCrx69Wpp2LBhsXI17ubw4cP2rBsAAEDlt9yofZ+ubbEppMbfMKgYAAC4XLjp0qWLvPvuu8VmT6nNNGfMmCFdu3a1d/0AAAAqt1tKhRg1oHjLli16nRu1eN8PP/ygW25KW7kYAADAqVtuWrVqJfv375fOnTtLz549dTfVww8/LNu2bdPTwgEAAKxUoa2r1a6c48aNs39tAAAAHBFuduzYUe4nbNOmzc3UBwAAoPLDTXh4uB44/HurEKtr8vPzb65GAAAAlR1ufvrpp5t5DQAAAOcKN40bN678mgAAAFi1ceZ7770nnTp1kvr16xetSjxnzhxZtWqVPeoEAADguHAzf/58iYuL05tmnjlzpmiMTY0aNXTAAQAAcKlwM3fuXFm4cKGeCu7l5VVUHhERITt37rR3/QAAACo33KjBxe3atStRrvaVUgv6AQAAuFS4adKkiWzfvr1EeUpKirRs2dJe9QIAAHDMCsVqvM2wYcPk0qVLet2btLQ0+ec//ylJSUny1ltvVawWAAAAVoWbIUOGiJ+fn4wfP15ycnKkX79+etbUq6++Kn379rVXvQAAABy3t1T//v31ocLNhQsXpE6dOhV7dQAAAKvH3Fy8eFGHGsXf31+fqynga9assXfdAAAAKj/c9OzZU9599139WK1zExkZKbNmzdLlag0cAAAAlwo3W7dulS5duujHH3zwgQQHB+tVilXgee211yqjjgAAAJUXblSXVPXq1fVj1RX18MMPi6enp9x1111FWzEAAAC4TLhp1qyZfPTRR3LkyBFZvXq13Hfffbo8KytLAgICKqOOAAAAlRduEhIS5IUXXpCQkBCJioqS6Ojoolac0lYuBgAAcOqp4H/5y1+kc+fOcuzYMWnbtm1Rebdu3aR37972rp9Lysn772airsyvqpd4eHhYXQ0AAByzzo0aRKyOa6lZU/iviClfiKuLaFxTVjwVTcABAJjfLYWyWzpUIDDFlsO/ycXLrt8CBQBwPxVquUFJqoVDtXS4eiBQXWomtDwBANwX4cbOAcffmz9SAACsRLcUAAAwSoWaGQ4ePKj3k9qzZ48+DwsLk2effVZCQ0PtXT8AAIDKbblRC/epMJOWliZt2rTRx+bNm+WOO+6QtWvX2vp0AAAA1rbcjB07VkaNGiXTpk0rUf7iiy/Kvffea8/6AQAAVG7LjeqKGjx4cInyJ554Qnbv3m3r0wEAAFgbbmrXri3bt28vUa7K6tSpY696AQAAOKZbaujQofLkk0/KoUOHpGPHjrps06ZNMn36dImLi6tYLQAAAKwKNxMmTJDq1avLrFmzJD4+XpfVr19fXn75ZRk5cqS96gUAAOCYbim1UJ0aUPzLL7/I2bNn9aEeq6ngFdmHaN68eXqHcV9fX73LuJqFdSNnzpyRYcOGSb169cTHx0eaN28un332mc2vCwAAzGRzuLnnnnt0wFBUC446lHPnzunv2WL58uW6K2vixImydetWvct4TEyMZGVllXp9Xl6eno31888/ywcffCD79u2ThQsXSoMGDWz9MQAAgKFs7pbasGGDDhnXu3Tpknz99dc2Pdfs2bP1GJ5Bgwbp8+TkZPn0009l8eLFemr59VT56dOn5ZtvvpGqVavqMtXqAwAAYHO42bFjR9FjNeU7MzOz6Dw/P19SUlJsakFRASk9Pb1o3I7i6ekp3bt3l9TU1FLv+fjjjyU6Olp3S61atUrP3OrXr59eX8fLy6vUe3Jzc/VRSLUwAQAAc5U73ISHh+sxNeoorfvJz89P5s6dW+4XPnnypA5FdevWLVauzvfu3VvqPWqG1vr166V///56nM2BAwfkmWeekcuXL+uurdIkJSXJpEmTyl0vAADgJuHmp59+koKCAmnatKke9KtaTQp5e3vrNW7Kaj2xl6tXr+rXWbBggX6tDh06yNGjR2XmzJllhhvVMnTtFHXVctOoUaNKrScAAHCBcNO4ceOigGEPQUFBOqAcP368WLk6Dw4OLvUeNUNKjbW5NkS1bNlSd5Gpbi4Vsq6nZlSpAwAAuAebZ0vZiwoiquVl3bp1RWUqOKlzNa6mNJ06ddJdUdcGrP379+vQU1qwAQAA7seycKOo7iI1lfudd97Re1Y9/fTTkp2dXTR7auDAgcUGHKvvq9lSak0dFWrUzKqpU6fqAcYAAAAVmgpuT3369JETJ05IQkKC7lpSg5bVrKvCQcYZGRl6BlUhNVZm9erVehHBNm3a6NlZKuio2VIAAACKR4EaJexG1IDiwMBAvbJyQECA1dVxOjl5VyQsYbV+vDsxRvy9Lc2/AADY/Pltc7eUmi116tSpEuVq1WL1PQAAACvZHG7U1gdqfZrrqYXy1LRsAAAAK5W7z0GtDlxIjXtRTUOFVNhRs5zYCgEAALhMuOnVq5f+qlYojo2NLfY9tfaMCjazZs2yfw0BAAAqI9wUri3TpEkT+e677/QifAAAAM7G5qkwahsGAAAAZ1Wheb5qfI06srKySmzHsHjxYnvVDQAAoPLDjdphOzExUSIiIvS2B2oMDgAAgMuGm+TkZFmyZIkMGDCgcmoEAADgyHVu1O7bHTt2vJnXBAAAcJ5wM2TIEFm2bFnl1AYAAMDR3VKXLl2SBQsWyBdffKE3r1Rr3Fxr9uzZN1snAAAAx4WbHTt26N27lV27dhX7HoOLAQCAy4WbL7/8snJqAgAAYMWYm0IHDhzQe0xdvHhRnxcUFNijPgAAAI4NN6dOnZJu3bpJ8+bNpUePHnLs2DFdPnjwYHn++edvrjYAAACODjejRo3Sg4gzMjLE39+/qLxPnz6SkpJys/UBAABw7JibNWvW6O6ohg0bFiu/7bbb5PDhwzdXGwAAAEe33GRnZxdrsSl0+vRp8fHxudn6AAAAODbcdOnSRd59991i07/V5pkzZsyQrl273lxtAAAAHN0tpUKMGlC8ZcsWvRXDmDFj5IcfftAtN5s2bbrZ+gAAADi25aZVq1ayf/9+6dy5s/Ts2VN3Uz388MOybds2CQ0NvbnaAAAAOLrlRgkMDJRx48bd7GsDAABY33Lz9ttvy4oVK0qUq7J33nnHXvUCAABwTLhJSkqSoKCgEuV16tSRqVOnVqwWAAAAVoUbtXhfkyZNSpQ3btxYfw8AAMClwo1qoVE7g1/v+++/l1tuucVe9QIAAHBMuHnsscdk5MiRenfw/Px8faxfv16effZZ6du3b8VqAQAAYNVsqcmTJ8vPP/+s17qpUuW/t6tF/AYOHMiYGwAA4FrhpqCgQDIzM2XJkiUyZcoU2b59u/j5+Unr1q31mBsAAACr2RxumjVrplckVhtlqgMAAMBlx9x4enrqQHPq1KnKqxEAAIAjBxRPmzZNRo8eLbt27bqZ1wUAAHCOAcVq4HBOTo60bdtWvL299Ziba6kNNAEAAFwm3MyZM6dyagIAAGBFuImNjbXH6wIAADjHmBvl4MGDMn78eL2gX1ZWli77/PPP9SwqAAAAlwo3X331lV7XZvPmzbJy5Uq5cOFC0fYLEydOrIw6AgAAVF64GTt2rF7Ab+3atXpAcaF77rlHvv32W1ufDgAAwNpws3PnTundu3epG2qePHnSXvUCAABwTLipUaOGHDt2rET5tm3bpEGDBhWrBQAAgFWzpdTO3y+++KKsWLFCPDw89KaZmzZtkhdeeEGvgQNz5OTli6vzq+ql/54CANyHzeFG7fw9bNgwadSokeTn50tYWJj+2q9fPz2DCuaImPKFuLqIxjVlxVPRBBwAcCMeBWo3zArIyMjQWzCo2VLt2rVzmU00z507J4GBgXL27FkJCAiwujpOR/11eDQ5VbYc/k1MsTsxRvy9bc7xAAAX/fyu8P/4t956q269Ufit2BzqvVQtHRcv57t8l5oJLU8AAAct4rdo0SJp1aqV+Pr66kM9fuutt6Si5s2bJyEhIfq5oqKiJC0trcxrlyxZoj+Arz3UfbAf9WeqWjpc+/Cy+o8RAGARm1tuEhISZPbs2TJixAiJjo7WZampqTJq1CjdVZWYmGjT8y1fvlzi4uIkOTlZBxu1d1VMTIzs27dPTy8vjWqOUt8vRMsRAACocLiZP3++LFy4UG+9UOihhx6SNm3a6MBja7hRQWno0KEyaNAgfa5CzqeffiqLFy/WCwaWRoWZ4OBgW6sOAADcgM3dUpcvX5aIiIgS5R06dJArV67Y9Fx5eXmSnp4u3bt3/1+FPD31uWoNKosaxNy4cWM95qdnz5433NMqNzdXD0K69gAAAOayOdwMGDBAt95cb8GCBdK/f3+bnkutaKymkdetW7dYuTrPzMws9Z7bb79dt+qsWrVKli5dqtfZ6dixo/zyyy+lXp+UlKRHVxcehYOgAQCAmapUdEDxmjVr5K677tLnahNNNd5GLeKnxs9c2+Vkb2qcT+FYH0UFm5YtW8qbb74pkydPLnF9fHx8sTqplhsCDgAA5rI53Ki1bdq3b68fHzx4UH8NCgrSh/qeLYN81T1eXl5y/PjxYuXqvLxjaqpWrarX2Tlw4ECp3/fx8dEHAABwDzaHmy+//NJuL652FVdjddatWye9evXSZaqbSZ0PHz68XM+hurXUZp49evSwW70AAIDrsnzZVtVlFBsbqwcpR0ZG6qng2dnZRbOnVFeX2pBTjZ1R1Gws1R3WrFkzOXPmjMycOVMOHz4sQ4YMsfgnAQAAzsDycNOnTx85ceKEXj9HDSIODw+XlJSUokHGaiyPmkFV6LffftNTx9W1NWvW1C0/33zzjd7jCgAAoMJ7S7kq9pZyDzl5VyQsYbV+zN5SAOBen98V2n4BAADAWRFuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABglCpWVwCobDl5+eLq/Kp6iYeHh9XVAACXQLiB8SKmfCGuLqJxTVnxVDQBBwDKgW4pGEm1dKhAYIoth3+Ti5ddvwUKAByBlhsYSbVwqJYOVw8EqkvNhJYnAHAkwg2MDjj+3vwVBwB3Q7cUAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEZxinAzb948CQkJEV9fX4mKipK0tLRy3ff+++/rzRF79epV6XUEAACuwfJws3z5comLi5OJEyfK1q1bpW3bthITEyNZWVk3vO/nn3+WF154Qbp06eKwugIAAOdnebiZPXu2DB06VAYNGiRhYWGSnJws/v7+snjx4jLvyc/Pl/79+8ukSZOkadOmDq0vAABwbpaGm7y8PElPT5fu3bv/r0Kenvo8NTW1zPsSExOlTp06Mnjw4N99jdzcXDl37lyxAwAAmMvScHPy5EndClO3bt1i5eo8MzOz1Hs2btwoixYtkoULF5brNZKSkiQwMLDoaNSokV3qDgAAnJPl3VK2OH/+vAwYMEAHm6CgoHLdEx8fL2fPni06jhw5Uun1BAAA1qli4WvrgOLl5SXHjx8vVq7Og4ODS1x/8OBBPZD4wQcfLCq7evWq/lqlShXZt2+fhIaGFrvHx8dHHwAAwD1Y2nLj7e0tHTp0kHXr1hULK+o8Ojq6xPUtWrSQnTt3yvbt24uOhx56SLp27aof0+UEAAAsbblR1DTw2NhYiYiIkMjISJkzZ45kZ2fr2VPKwIEDpUGDBnrsjFoHp1WrVsXur1Gjhv56fTkAAHBPloebPn36yIkTJyQhIUEPIg4PD5eUlJSiQcYZGRl6BhUAAEB5eBQUFBSIG1FTwdWsKTW4OCAgwOrqADeUk3dFwhJW68e7E2PE39vy30cAwOk/v2kSAQAARuHXQMBF5OTli6vzq+ql94MDgMpEuAFcRMSUL8TVRTSuKSueiibgAKhUdEsBTt7SoQKBKbYc/k0uXnb9FigAzo2WG8CJqRYO1dLh6oFAdamZ0PIEwDUQbgAXCDjMkgKA8qNbCgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADBKFasrAMC95OTli6vzq+olHh4eVlcDQBkINwAcKmLKF+LqIhrXlBVPRRNwACdFtxQAh7R0qEBgii2Hf5OLl12/BQowFS03ACqdauFQLR2uHghUl5oJLU+A6Qg3ABwWcPy9+S8HQOWjWwoAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMAoragFABbABKOC8CDcAUAEmbMPABqAwFd1SAFBObAAKuAZabgCgnNgAFHANhBsAsAEbgALOj24pAABgFMINAAAwCuEGAAAYhY5jAHBjJqzXo7BmD65FuAEAN2bKrCnW7IHTdUvNmzdPQkJCxNfXV6KioiQtLa3Ma1euXCkRERFSo0YNqVatmoSHh8t7773n0PoCgCszbb0ehTV74FQtN8uXL5e4uDhJTk7WwWbOnDkSExMj+/btkzp16pS4vlatWjJu3Dhp0aKFeHt7yyeffCKDBg3S16r7AADusV7P9Wv2mNDFRveafXgUFBQUiIVUoLnzzjvl9ddf1+dXr16VRo0ayYgRI2Ts2LHleo727dvLAw88IJMnT/7da8+dOyeBgYFy9uxZCQgIuOn6AwCsk5N3RcISVospwuoF/H/3mrg8ewc1Wz6/LW25ycvLk/T0dImPjy8q8/T0lO7du0tqaurv3q9y2fr163Urz/Tp00u9Jjc3Vx/X/uEAAMzqYlPdUibYfeyc3DHRjLC2OzHGsgUvLQ03J0+elPz8fKlbt26xcnW+d+/eMu9Tqa1BgwY6tHh5eckbb7wh9957b6nXJiUlyaRJk+xedwCA9UzpYlN9KI8mp+pwAwPG3FRE9erVZfv27XLhwgVZt26dHrPTtGlTufvuu0tcq1qF1PevbblR3V4AADOYsiXGpyM7u3xIu75VzSqW/m0ICgrSLS/Hjx8vVq7Og4ODy7xPdV01a9ZMP1azpfbs2aNbaEoLNz4+PvoAAMCZmRLSxN2ngqvZTh06dNCtL4XUgGJ1Hh0dXe7nUfdcO64GAAC4L8sjouoyio2N1WvXREZG6qng2dnZenq3MnDgQD2+RrXMKOqrujY0NFQHms8++0yvczN//nyLfxIAAOAMLA83ffr0kRMnTkhCQoJkZmbqbqaUlJSiQcYZGRm6G6qQCj7PPPOM/PLLL+Ln56fXu1m6dKl+HgAAAMvXuXE01rkBAMDsz2+n2H4BAADAXgg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRLN9+wdEKF2RWKx0CAADXUPi5XZ6NFdwu3Jw/f15/bdSokdVVAQAAFfgcV9sw3Ijb7S119epV+fXXX6V69eri4eFh91SpQtORI0fYt8oJ8H44F94P58L74Xx4T25MxRUVbOrXr19sQ+3SuF3LjfoDadiwYaW+hvpLyV9M58H74Vx4P5wL74fz4T0p2++12BRiQDEAADAK4QYAABiFcGNHPj4+MnHiRP0V1uP9cC68H86F98P58J7Yj9sNKAYAAGaj5QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbuxk3rx5EhISIr6+vhIVFSVpaWlWV8ltJSUlyZ133qlXoa5Tp4706tVL9u3bZ3W18P+mTZumVwd/7rnnrK6K2zp69Kg8/vjjcsstt4ifn5+0bt1atmzZYnW13FJ+fr5MmDBBmjRpot+L0NBQmTx5crn2T0LZCDd2sHz5comLi9NT+LZu3Spt27aVmJgYycrKsrpqbumrr76SYcOGybfffitr166Vy5cvy3333SfZ2dlWV83tfffdd/Lmm29KmzZtrK6K2/rtt9+kU6dOUrVqVfn8889l9+7dMmvWLKlZs6bVVXNL06dPl/nz58vrr78ue/bs0eczZsyQuXPnWl01l8ZUcDtQLTWqpUD95Szcv0rtDzJixAgZO3as1dVzeydOnNAtOCr0/PGPf7S6Om7rwoUL0r59e3njjTdkypQpEh4eLnPmzLG6Wm5H/Z+0adMm+frrr62uCkTkz3/+s9StW1cWLVpUVPbII4/oVpylS5daWjdXRsvNTcrLy5P09HTp3r17sf2r1HlqaqqldcN/nT17Vn+tVauW1VVxa6o17YEHHij2bwWO9/HHH0tERIQ8+uijOvS3a9dOFi5caHW13FbHjh1l3bp1sn//fn3+/fffy8aNG+X++++3umouze02zrS3kydP6j5Tlbyvpc737t1rWb0gRa1oamyHaoZv1aqV1dVxW++//77uslXdUrDWoUOHdDeI6kp/6aWX9HsycuRI8fb2ltjYWKur55YtaWo38BYtWoiXl5f+PHnllVekf//+VlfNpRFuYHxrwa5du/RvQrDGkSNH5Nlnn9Xjn9SAe1gf+FXLzdSpU/W5arlR/0aSk5MJNxb417/+Jf/4xz9k2bJlcscdd8j27dv1L2T169fn/bgJhJubFBQUpNP28ePHi5Wr8+DgYMvqBZHhw4fLJ598Iv/5z3+kYcOGVlfHbaluWzW4Xo23KaR+O1Xvixqnlpubq/8NwTHq1asnYWFhxcpatmwpH374oWV1cmejR4/WrTd9+/bV52rm2uHDh/WsT8JNxTHm5iapptwOHTroPtNrfzNS59HR0ZbWzV2pMfIq2Pz73/+W9evX6ymWsE63bt1k586d+jfSwkO1HKhmd/WYYONYqov2+qUR1HiPxo0bW1Ynd5aTk6PHaV5L/ZtQnyOoOFpu7ED1XauErf7DjoyM1DNA1LTjQYMGWV01t+2KUk28q1at0mvdZGZm6vLAwEA9AwGOpd6D68c7VatWTa+xwjgoxxs1apQexKq6pf7617/qNbkWLFigDzjegw8+qMfY3Hrrrbpbatu2bTJ79mx54oknrK6aS2MquJ2o5vWZM2fqD1I1xfW1117TU8TheGqBuNK8/fbb8re//c3h9UFJd999N1PBLaS6a+Pj4+XHH3/ULZvqF7ShQ4daXS23dP78eb2In2ppVt23aqzNY489JgkJCbpnABVDuAEAAEZhzA0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQCnp9YaffLJJ6VWrVp6BWq1J9WNbNiwQV935syZMq9ZsmSJ1KhRoxJqC8Bq7C0FwOmlpKToMKJCS9OmTSUoKMjqKgFwYoQbAE7v4MGDUq9ePb3hIwD8HrqlADg1tdnpiBEjJCMjQ3c1hYSESG5urowcOVLq1Kkjvr6+0rlzZ/nuu+9u+Dyq5UftvOzv7y+9e/eWU6dOOexnAOBYhBsATu3VV1+VxMREadiwoRw7dkyHmDFjxsiHH34o77zzjmzdulWaNWsmMTExcvr06VKfY/PmzTJ48GAZPny4Hq/TtWtXmTJlisN/FgCOQbgB4NQCAwOlevXq4uXlJcHBwbrlZf78+TJz5ky5//77JSwsTBYuXCh+fn6yaNGiMgPSn/70Jx2Kmjdvrlt9VBgCYCbCDQCXG39z+fJl6dSpU1FZ1apVJTIyUvbs2VPqPao8KiqqWFl0dHSl1xWANQg3AADAKIQbAC4lNDRUvL29ZdOmTUVlqiVHjcVRXVSladmypR53c61vv/220usKwBpMBQfgUqpVqyZPP/20jB49Wi/qp2ZAzZgxQ3JycvSg4dKoMTaqG+vvf/+79OzZU1avXq3XzgFgJlpuALicadOmySOPPCIDBgyQ9u3by4EDB3RgqVmzZqnX33XXXXrQsRpY3LZtW1mzZo2MHz/e4fUG4BgeBWpdcwAAAEPQcgMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAMcn/AaHxoHr//PSaAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tpot\n",
    "\n",
    "selection_evaluation_pruning = [.9, .3]\n",
    "selection_evaluation_scaling = .2\n",
    "\n",
    "#Population and budget use stepwise\n",
    "fig, ax1 = plt.subplots()\n",
    "\n",
    "interpolated_values = tpot.utils.beta_interpolation(start=selection_evaluation_pruning[0], end=selection_evaluation_pruning[-1], n=cv, n_steps=cv, scale=selection_evaluation_scaling)\n",
    "ax1.step(list(range(len(interpolated_values))), interpolated_values, label=f\"threshold\")\n",
    "ax1.set_xlabel(\"fold\")\n",
    "ax1.set_ylabel(\"percent to select\")\n",
    "#ax1.legend(loc='center left', bbox_to_anchor=(1.1, 0.4))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  10%|█         | 1/10 [02:23<21:31, 143.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  1\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  20%|██        | 2/10 [04:00<15:30, 116.31s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  2\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  30%|███       | 3/10 [05:42<12:48, 109.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  3\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  40%|████      | 4/10 [07:36<11:08, 111.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  4\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  50%|█████     | 5/10 [09:12<08:48, 105.72s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  5\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  60%|██████    | 6/10 [11:04<07:11, 107.81s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  6\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  70%|███████   | 7/10 [12:54<05:26, 108.71s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  7\n",
      "Best roc_auc_score score: 0.9212394545585602\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  80%|████████  | 8/10 [14:45<03:38, 109.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  8\n",
      "Best roc_auc_score score: 0.925549420935039\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  90%|█████████ | 9/10 [16:49<01:54, 114.03s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  9\n",
      "Best roc_auc_score score: 0.925549420935039\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation: 100%|██████████| 10/10 [18:36<00:00, 111.67s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  10\n",
      "Best roc_auc_score score: 0.925549420935039\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/decomposition/_fastica.py:595: UserWarning: n_components is too large: it will be set to 20\n",
      "  warnings.warn(\n",
      "/opt/anaconda3/envs/tpotenv/lib/python3.10/site-packages/sklearn/decomposition/_fastica.py:128: ConvergenceWarning: FastICA did not converge. Consider increasing tolerance or the maximum number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total time: 1129.1526980400085\n",
      "test score:  0.9324219154371735\n"
     ]
    }
   ],
   "source": [
    "est = tpot.TPOTEstimator(  \n",
    "                            generations=10,\n",
    "                            max_time_mins=None,\n",
    "                            scorers=['roc_auc_ovr'],\n",
    "                            scorers_weights=[1],\n",
    "                            classification=True,\n",
    "                            search_space = search_space,\n",
    "                            population_size=100,\n",
    "                            n_jobs=32,\n",
    "                            cv=cv,\n",
    "                            verbose=3,\n",
    "                            random_state=42,\n",
    "\n",
    "                            selection_evaluation_pruning  = selection_evaluation_pruning,\n",
    "                            selection_evaluation_scaling = selection_evaluation_scaling,\n",
    "                            )\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "est.fit(X_train, y_train)\n",
    "print(f\"total time: {time.time()-start}\")\n",
    "print(\"test score: \", scorer(est, X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>roc_auc_score</th>\n",
       "      <th>Parents</th>\n",
       "      <th>Variation_Function</th>\n",
       "      <th>Individual</th>\n",
       "      <th>Generation</th>\n",
       "      <th>roc_auc_score_step_0</th>\n",
       "      <th>Submitted Timestamp</th>\n",
       "      <th>Completed Timestamp</th>\n",
       "      <th>Eval Error</th>\n",
       "      <th>roc_auc_score_step_1</th>\n",
       "      <th>roc_auc_score_step_2</th>\n",
       "      <th>roc_auc_score_step_3</th>\n",
       "      <th>roc_auc_score_step_4</th>\n",
       "      <th>roc_auc_score_step_5</th>\n",
       "      <th>roc_auc_score_step_6</th>\n",
       "      <th>roc_auc_score_step_7</th>\n",
       "      <th>roc_auc_score_step_8</th>\n",
       "      <th>roc_auc_score_step_9</th>\n",
       "      <th>Pareto_Front</th>\n",
       "      <th>Instance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.812263</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.811153</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.799213</td>\n",
       "      <td>0.807710</td>\n",
       "      <td>0.813587</td>\n",
       "      <td>0.797528</td>\n",
       "      <td>0.820692</td>\n",
       "      <td>0.827614</td>\n",
       "      <td>0.815069</td>\n",
       "      <td>0.805447</td>\n",
       "      <td>0.824616</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(MinMaxScaler(), RFE(estimator=ExtraTreesClass...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.848068</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.846478</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.839894</td>\n",
       "      <td>0.844619</td>\n",
       "      <td>0.848321</td>\n",
       "      <td>0.846915</td>\n",
       "      <td>0.857902</td>\n",
       "      <td>0.855875</td>\n",
       "      <td>0.827655</td>\n",
       "      <td>0.850938</td>\n",
       "      <td>0.862081</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(Passthrough(), RFE(estimator=ExtraTreesClassi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.831502</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.817219</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.827888</td>\n",
       "      <td>0.821911</td>\n",
       "      <td>0.825558</td>\n",
       "      <td>0.830020</td>\n",
       "      <td>0.831529</td>\n",
       "      <td>0.836955</td>\n",
       "      <td>0.844634</td>\n",
       "      <td>0.832499</td>\n",
       "      <td>0.846805</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(StandardScaler(), VarianceThreshold(threshold...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.830374</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.817150</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.831885</td>\n",
       "      <td>0.820694</td>\n",
       "      <td>0.824899</td>\n",
       "      <td>0.824409</td>\n",
       "      <td>0.827861</td>\n",
       "      <td>0.833923</td>\n",
       "      <td>0.844308</td>\n",
       "      <td>0.832798</td>\n",
       "      <td>0.845818</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(MinMaxScaler(), SelectFromModel(estimator=Ext...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.850091</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.843524</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>1.740197e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.841176</td>\n",
       "      <td>0.840619</td>\n",
       "      <td>0.846209</td>\n",
       "      <td>0.849561</td>\n",
       "      <td>0.854367</td>\n",
       "      <td>0.858035</td>\n",
       "      <td>0.860165</td>\n",
       "      <td>0.845179</td>\n",
       "      <td>0.862077</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(Normalizer(norm='max'), SelectFwe(alpha=0.000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>983</th>\n",
       "      <td>0.886974</td>\n",
       "      <td>(13, 13)</td>\n",
       "      <td>ind_mutate</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.871580</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.887762</td>\n",
       "      <td>0.882504</td>\n",
       "      <td>0.860872</td>\n",
       "      <td>0.898100</td>\n",
       "      <td>0.885523</td>\n",
       "      <td>0.893527</td>\n",
       "      <td>0.904779</td>\n",
       "      <td>0.884557</td>\n",
       "      <td>0.900537</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(StandardScaler(), SelectFromModel(estimator=E...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>986</th>\n",
       "      <td>0.850281</td>\n",
       "      <td>(35, 470)</td>\n",
       "      <td>ind_crossover</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.837493</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.858289</td>\n",
       "      <td>0.844141</td>\n",
       "      <td>0.851260</td>\n",
       "      <td>0.848909</td>\n",
       "      <td>0.853002</td>\n",
       "      <td>0.856132</td>\n",
       "      <td>0.845356</td>\n",
       "      <td>0.847830</td>\n",
       "      <td>0.860393</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(StandardScaler(), SelectPercentile(percentile...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>990</th>\n",
       "      <td>0.878811</td>\n",
       "      <td>(866, 866)</td>\n",
       "      <td>ind_mutate</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.875842</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.862567</td>\n",
       "      <td>0.881858</td>\n",
       "      <td>0.885539</td>\n",
       "      <td>0.874347</td>\n",
       "      <td>0.888858</td>\n",
       "      <td>0.891205</td>\n",
       "      <td>0.882103</td>\n",
       "      <td>0.863952</td>\n",
       "      <td>0.881838</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(Normalizer(norm='l1'), SelectPercentile(perce...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>991</th>\n",
       "      <td>0.835669</td>\n",
       "      <td>(72, 855)</td>\n",
       "      <td>ind_crossover</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.838375</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.844572</td>\n",
       "      <td>0.837234</td>\n",
       "      <td>0.822799</td>\n",
       "      <td>0.818868</td>\n",
       "      <td>0.840971</td>\n",
       "      <td>0.845122</td>\n",
       "      <td>0.816390</td>\n",
       "      <td>0.840709</td>\n",
       "      <td>0.851650</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(MinMaxScaler(), SelectPercentile(percentile=4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>992</th>\n",
       "      <td>0.892459</td>\n",
       "      <td>(898, 898)</td>\n",
       "      <td>ind_mutate</td>\n",
       "      <td>&lt;tpot.search_spaces.pipelines.sequential.Seque...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.881991</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>1.740198e+09</td>\n",
       "      <td>None</td>\n",
       "      <td>0.893987</td>\n",
       "      <td>0.882514</td>\n",
       "      <td>0.887394</td>\n",
       "      <td>0.902290</td>\n",
       "      <td>0.894360</td>\n",
       "      <td>0.903944</td>\n",
       "      <td>0.884672</td>\n",
       "      <td>0.889588</td>\n",
       "      <td>0.903849</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(RobustScaler(quantile_range=(0.0911728428421,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>326 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     roc_auc_score     Parents Variation_Function  \\\n",
       "0         0.812263         NaN                NaN   \n",
       "1         0.848068         NaN                NaN   \n",
       "4         0.831502         NaN                NaN   \n",
       "5         0.830374         NaN                NaN   \n",
       "6         0.850091         NaN                NaN   \n",
       "..             ...         ...                ...   \n",
       "983       0.886974    (13, 13)         ind_mutate   \n",
       "986       0.850281   (35, 470)      ind_crossover   \n",
       "990       0.878811  (866, 866)         ind_mutate   \n",
       "991       0.835669   (72, 855)      ind_crossover   \n",
       "992       0.892459  (898, 898)         ind_mutate   \n",
       "\n",
       "                                            Individual  Generation  \\\n",
       "0    <tpot.search_spaces.pipelines.sequential.Seque...         0.0   \n",
       "1    <tpot.search_spaces.pipelines.sequential.Seque...         0.0   \n",
       "4    <tpot.search_spaces.pipelines.sequential.Seque...         0.0   \n",
       "5    <tpot.search_spaces.pipelines.sequential.Seque...         0.0   \n",
       "6    <tpot.search_spaces.pipelines.sequential.Seque...         0.0   \n",
       "..                                                 ...         ...   \n",
       "983  <tpot.search_spaces.pipelines.sequential.Seque...         9.0   \n",
       "986  <tpot.search_spaces.pipelines.sequential.Seque...         9.0   \n",
       "990  <tpot.search_spaces.pipelines.sequential.Seque...         9.0   \n",
       "991  <tpot.search_spaces.pipelines.sequential.Seque...         9.0   \n",
       "992  <tpot.search_spaces.pipelines.sequential.Seque...         9.0   \n",
       "\n",
       "     roc_auc_score_step_0  Submitted Timestamp  Completed Timestamp  \\\n",
       "0                0.811153         1.740198e+09         1.740198e+09   \n",
       "1                0.846478         1.740197e+09         1.740197e+09   \n",
       "4                0.817219         1.740197e+09         1.740197e+09   \n",
       "5                0.817150         1.740197e+09         1.740197e+09   \n",
       "6                0.843524         1.740197e+09         1.740197e+09   \n",
       "..                    ...                  ...                  ...   \n",
       "983              0.871580         1.740198e+09         1.740198e+09   \n",
       "986              0.837493         1.740198e+09         1.740198e+09   \n",
       "990              0.875842         1.740198e+09         1.740198e+09   \n",
       "991              0.838375         1.740198e+09         1.740198e+09   \n",
       "992              0.881991         1.740198e+09         1.740198e+09   \n",
       "\n",
       "    Eval Error  roc_auc_score_step_1  roc_auc_score_step_2  \\\n",
       "0         None              0.799213              0.807710   \n",
       "1         None              0.839894              0.844619   \n",
       "4         None              0.827888              0.821911   \n",
       "5         None              0.831885              0.820694   \n",
       "6         None              0.841176              0.840619   \n",
       "..         ...                   ...                   ...   \n",
       "983       None              0.887762              0.882504   \n",
       "986       None              0.858289              0.844141   \n",
       "990       None              0.862567              0.881858   \n",
       "991       None              0.844572              0.837234   \n",
       "992       None              0.893987              0.882514   \n",
       "\n",
       "     roc_auc_score_step_3  roc_auc_score_step_4  roc_auc_score_step_5  \\\n",
       "0                0.813587              0.797528              0.820692   \n",
       "1                0.848321              0.846915              0.857902   \n",
       "4                0.825558              0.830020              0.831529   \n",
       "5                0.824899              0.824409              0.827861   \n",
       "6                0.846209              0.849561              0.854367   \n",
       "..                    ...                   ...                   ...   \n",
       "983              0.860872              0.898100              0.885523   \n",
       "986              0.851260              0.848909              0.853002   \n",
       "990              0.885539              0.874347              0.888858   \n",
       "991              0.822799              0.818868              0.840971   \n",
       "992              0.887394              0.902290              0.894360   \n",
       "\n",
       "     roc_auc_score_step_6  roc_auc_score_step_7  roc_auc_score_step_8  \\\n",
       "0                0.827614              0.815069              0.805447   \n",
       "1                0.855875              0.827655              0.850938   \n",
       "4                0.836955              0.844634              0.832499   \n",
       "5                0.833923              0.844308              0.832798   \n",
       "6                0.858035              0.860165              0.845179   \n",
       "..                    ...                   ...                   ...   \n",
       "983              0.893527              0.904779              0.884557   \n",
       "986              0.856132              0.845356              0.847830   \n",
       "990              0.891205              0.882103              0.863952   \n",
       "991              0.845122              0.816390              0.840709   \n",
       "992              0.903944              0.884672              0.889588   \n",
       "\n",
       "     roc_auc_score_step_9  Pareto_Front  \\\n",
       "0                0.824616           NaN   \n",
       "1                0.862081           NaN   \n",
       "4                0.846805           NaN   \n",
       "5                0.845818           NaN   \n",
       "6                0.862077           NaN   \n",
       "..                    ...           ...   \n",
       "983              0.900537           NaN   \n",
       "986              0.860393           NaN   \n",
       "990              0.881838           NaN   \n",
       "991              0.851650           NaN   \n",
       "992              0.903849           NaN   \n",
       "\n",
       "                                              Instance  \n",
       "0    (MinMaxScaler(), RFE(estimator=ExtraTreesClass...  \n",
       "1    (Passthrough(), RFE(estimator=ExtraTreesClassi...  \n",
       "4    (StandardScaler(), VarianceThreshold(threshold...  \n",
       "5    (MinMaxScaler(), SelectFromModel(estimator=Ext...  \n",
       "6    (Normalizer(norm='max'), SelectFwe(alpha=0.000...  \n",
       "..                                                 ...  \n",
       "983  (StandardScaler(), SelectFromModel(estimator=E...  \n",
       "986  (StandardScaler(), SelectPercentile(percentile...  \n",
       "990  (Normalizer(norm='l1'), SelectPercentile(perce...  \n",
       "991  (MinMaxScaler(), SelectPercentile(percentile=4...  \n",
       "992  (RobustScaler(quantile_range=(0.0911728428421,...  \n",
       "\n",
       "[326 rows x 20 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est.evaluated_individuals[est.evaluated_individuals['roc_auc_score_step_9']>0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the above methods can be used independently or simultaneously as done below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  10%|█         | 1/10 [01:34<14:09, 94.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  1\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  20%|██        | 2/10 [02:26<09:14, 69.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  2\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  30%|███       | 3/10 [03:41<08:23, 71.97s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  3\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  40%|████      | 4/10 [04:52<07:09, 71.53s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  4\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  50%|█████     | 5/10 [05:52<05:37, 67.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  5\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  60%|██████    | 6/10 [07:13<04:48, 72.10s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  6\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  70%|███████   | 7/10 [08:06<03:17, 65.84s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  7\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  80%|████████  | 8/10 [08:57<02:02, 61.13s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  8\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation:  90%|█████████ | 9/10 [09:39<00:55, 55.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  9\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generation: 100%|██████████| 10/10 [10:17<00:00, 61.70s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation:  10\n",
      "Best roc_auc_score score: 0.8515086951804098\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total time: 621.607882976532\n",
      "test score:  0.9084772293865335\n"
     ]
    }
   ],
   "source": [
    "est = tpot.TPOTEstimator(  \n",
    "                            generations=10,\n",
    "                            max_time_mins=None,\n",
    "                            scorers=['roc_auc_ovr'],\n",
    "                            scorers_weights=[1],\n",
    "                            classification=True,\n",
    "                            search_space = search_space,\n",
    "                            population_size=30,\n",
    "                            n_jobs=3,\n",
    "                            cv=cv,\n",
    "                            verbose=3,\n",
    "\n",
    "                            initial_population_size=initial_population_size,\n",
    "                            population_scaling = population_scaling,\n",
    "                            generations_until_end_population = generations_until_end_population,\n",
    "                            \n",
    "                            budget_range = budget_range,\n",
    "                            generations_until_end_budget=generations_until_end_budget,\n",
    "                            \n",
    "                            threshold_evaluation_pruning = threshold_evaluation_pruning,\n",
    "                            threshold_evaluation_scaling = threshold_evaluation_scaling,\n",
    "\n",
    "                            selection_evaluation_pruning  = selection_evaluation_pruning,\n",
    "                            selection_evaluation_scaling = selection_evaluation_scaling,\n",
    "                            )\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "est.fit(X_train, y_train)\n",
    "print(f\"total time: {time.time()-start}\")\n",
    "print(\"test score: \", scorer(est, X_test, y_test))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tpotenv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
